Aura H2O Bulk editing Pocket articles

[ceridwen]

I have a bunch of Pocket articles that I have downloaded to my Kobo at some point and then archived using another Pocket client, usually because the article was broken on the Kobo. Unfortunately, archiving an article with one of the other clients still leaves the article downloaded on my Kobo, and they're now consuming a couple of hundred megabytes of space. I want to delete the files on my Kobo without deleting them from my Pocket account, and without going one-by-one through my 800+ archived articles.

The files for the articles live in .kobo/articles, as far as I can tell. Is there metadata stored somewhere else? I'm probably going to have to write a script that uses the Pocket API to figure out what articles are archived and then delete the files, but I'm not sure about the best way to connect the files for an article and the article metadata retrieved from the API. I also want to know, before deleting the files directly, if there are any other edits I need to make elsewhere to avoid breaking the software on the Kobo that handles Pocket articles.

I tried some searches on this forum, but I couldn't find any technical information on the Kobo Pocket software or file layout.

[davidfor]

The metadata is stored in the database. You could run SQL against this to do the deletes or otherwise mark them as archived or deleted. I haven't paid much attention to the Pocket articles, but quickly:
- The table "content" has two rows for each article - one describes the article, the other is probably used for the current reading position.
- The contentId of the main row is the directory name of the article.
- The rows all have MimeType = 'application/x-kobo-html+pocket'
- There is a "PocketStatus". Changing that is probably what is needed, but I don't know the values.

But, I think the answer is a lot simpler: Sign out of Pocket on the device. This should clear all the articles. Then sign in and it should only download the current articles.

If you don't trust it, take a full copy of the book partition before doing the above. If it doesn't work, restore the copy in place of what you have. And I'd recommend doing this before fiddling with the database.

[ceridwen]

Thanks!

Pocket on Kobo does have an option to delete all articles, and I thought about using that, but I expected then that I would have redownload by hand all the articles I've saved but haven't read yet. That list is shorter but still not short enough. Have you tried signing in and out before, and only articles that weren't archived were downloaded again?

And yeah, I'm definitely going to backup the whole partition, including the database, before I do anything. Is there a recommended way of doing this beyond just recursively copying all the files?

[davidfor]

Originally Posted by ceridwen

Thanks!

Pocket on Kobo does have an option to delete all articles, and I thought about using that, but I expected then that I would have redownload by hand all the articles I've saved but haven't read yet. That list is shorter but still not short enough. Have you tried signing in and out before, and only articles that weren't archived were downloaded again?

I haven't explicitly signed in and out. But, I have several devices and I don't always have Pocket configured on them all. When I do, I usually copy the configuration from one of the other devices. Your question has reminded me that I don't always get the articles back, but I've also realised I've been making a mistake in copying the configuration.

The Pocket configuration is in the "Kobo Reader.conf" file and looks like:

[Pocket]
AccessToken=xxxxxxx-xxxx-xxxx-xxxx-xxxxxx
LastSync=1483795204
UnsyncedUrls=@Invalid()
Username=pocketusername

I have never seen the "UnsyncedUrls" with a real value. I assume it is supposed to be a list of URLs that Pocket understand and have not been updated. I suspect it is not actually used.

"LastSync" is the timestamp (a Unix epoch time) of when the last sync to Pocket was done. As I am copying the configuration from my Glo HD, the other devices are getting a recent timestamp and hence only downloading articles added since that timestamp. Which will frequently be none. I'm not sure of this, but it agrees with what I have seen. I'll do some experiments at home tonight.

If I am right, you could easily reduce the articles to those added since a particular date. Otherwise, I think you would need to archive them or mark them as read on the Pocket site to prevent them from downloading.

And yeah, I'm definitely going to backup the whole partition, including the database, before I do anything. Is there a recommended way of doing this beyond just recursively copying all the files?

I just select all the files and directories in the root of the device and drag them to a folder on my PC. If you are using a Mac or Linux machine, you need to make sure you get the hidden directories. These have names starting with a dot. ".kobo" is the most important one.

[davidfor]

I've had a play and the results agree with what I thought:

• Signing out of of Pocket and back in downloads all the articles in "My List".
• Signing out of Pocket and copying the configuration from another device, only downloaded those added since the last sync.
• Changing the "LastSync" to some time in the past downloaded all articles added since that date.
• The list of Archived articles was downloaded, but the articles were not.

I haven't done a lot of tests, but this agrees with what I have seen before.

I'm not sure how much that helps. Fiddling with the timestamp will reduce the articles downloaded. Whether that prunes them them to a suitable amount, I don't know.

[compurandom]

I've played with the pocket stuff a bit. After a reset, it does sync only a portion of your articles, and the ones since the last sync would make sense. I usually cherry pick a few more than what it automatically syncs by clicking on the download button in the list, and then change the library option so it only shows what is already downloaded.

You can also delete articles from the kobo manually without deleting them from pocket, and it won't automatically resync them.

I think the easiest way to remove articles in bulk would be to set the option to not download archived articles and then in the web interface archive the ones you don't want on the kobo. I think this causes the kobo to remove them, but I'm not 100% sure.

I'm a bit sad that Pocket has updated the android client so much but the kobo client has not gotten a recent update. It really would be nice to have tagging and bulk editing on the kobo.

[ceridwen]

Originally Posted by davidfor

I've had a play and the results agree with what I thought:

• Signing out of of Pocket and back in downloads all the articles in "My List".
  ...

I haven't done a lot of tests, but this agrees with what I have seen before.

I'm not sure how much that helps. Fiddling with the timestamp will reduce the articles downloaded. Whether that prunes them them to a suitable amount, I don't know.

Thanks. I think that getting rid of all the archived-but-not-deleted articles will be good enough for my purposes. I'm going to try this sometime this weekend, I'll report back.

I do wish they'd update the Kobo client, though I'd imagine that probably hasn't happened because of lack of coordination between Kobo and Pocket.

[ceridwen]

Well, I tried logging in and logging out. It only downloaded a tiny fraction of all the unread articles I have in Pocket, 50. It doesn't even include all the articles that I have saved in either My List or Archive, downloaded or otherwise. AFAICT, at this point I have no choice but to either restore from backup or write the script to do the editing. Unfortunately, I don't know how to get it download files the rest of the files, or even include the entries, and I'm worried about breaking something by restoring from backup. Since I'm going to encounter this problem again in the future (any time I have an article downloaded on my ereader that I then read on another device, which is pretty often), I'm probably just going to write the script. I assume that by editing the sqlite db, I can probably restore the missing My List/Archive entries. Is there anyway to force it to download the rest of the articles it's missing?

[davidfor]

Before doing anything else, try editing the config file and removing the "LastSync" and "UnsyncedUrls" entries and then syncing again. It is possible that the device is only downloading a set number of articles. Removing these entries might let it download more. Alternatively, setting the timestamp for LastSync to something a long time ago might do it as well.

I doubt if editing the current database will get more articles downloaded. I think you would need to create entries for the missing articles. Without knowing what the articles where and the values needed, this would be impossible. You might be able to copy them from the backup database but it would take some experimenting to get it right.

For restoring the backup, I've done this plenty of times with no problems. Just remove all the files on the device and copy the backup files to the device. Then restart it after ejecting.

[compurandom]

Originally Posted by ceridwen

Well, I tried logging in and logging out. It only downloaded a tiny fraction of all the unread articles I have in Pocket,

Is there anyway to force it to download the rest of the articles it's missing?

Yes, there is, if you don't mind selecting them one at a time.
I've done this several times, it typically isn't all that painful as it queues them up for download in the background.

In the article screen , tap the three bar icon in the pocket header bar (below the three bar icon at the top of the screen). Tap library settings.

In library settings, uncheck "Only show downloaded items" and close the settings to go back to the article screen.

Then you can go through all your articles and tap download on anything you want on your ebook. Similarly, you can change selection to Archived and go through that list too.

Also, even with that library option checked, if you search your library, pocket articles can be found and you can queue them for downloading there too.

[ceridwen]

Originally Posted by davidfor

Before doing anything else, try editing the config file and removing the "LastSync" and "UnsyncedUrls" entries and then syncing again. It is possible that the device is only downloading a set number of articles. Removing these entries might let it download more. Alternatively, setting the timestamp for LastSync to something a long time ago might do it as well.

Unfortunately, changing the LastSync and UnsyncedUrls don't seem to do anything. I've never seen any more articles downloaded, or the articles that are in my Pocket account but aren't on my Kobo added to my Kobo's list.

Originally Posted by compurandom

Yes, there is, if you don't mind selecting them one at a time.

This is exactly what I don't want to do.

I'm pretty sure I have no choice but to write a script that does the synchronization, deleting articles off my Kobo that I've read and archived elsewhere and downloading the articles that it's not synchronizing.

Does anyone know if it's possible to set an article as pending in the SQL database so that the Kobo will download it on a synch? If not, is there any other way to get the Kobo to download an article by editing the database?