Libra 2 Restored backup is reset again after next "Importing content" Calibre sync, any idea?

[Xylon]

Hi all,

I'm trying to restore a backup created by KoboUtilities, but I'm running into an issue where the next "Importing content" after a Calibre sync resets all progress. Basically I've done the following:

1. Factory reset the device
2. Copied the books back with Calibre. See unread status on all books.
3. Copy database from backup to device. Disconnect it from explorer. It now shows the old progress on all books and all the highlights. Restarting also keeps the progress. Exactly what I want.
4. Copy books from Calibre (only the ones that weren't in the backupped database, so not the ones that are showing progress).
5. After I disconnect it and the ereader does the "Importing content", I see the books, one by one, resetting to 0% progress and unread.

Any idea how I can keep the progress on the books after the backup, and after adding new books using Calibre? Did I do something in the wrong order?

Thank you.

[DNSB]

The only way I can duplicate your results to to send the books that were already on the Kobo to the Kobo after the database restore.

After a factory reset, I will use the Kobo Utlities store/restore current bookmark to reset the reading status, etc. I also use the Update metadata in Device Library to set the Data Added to the Date the book was added to calibre.

[Xylon]

Interesting, thank you.

So to make the initial example a bit clearer:

1. my case for example I've copied books A, B, C, D and E to the ereader. All stats were set to unread automatically.
2. I then copied the sqlite file, which had information about A, B and C (D and E were added later and were not in this backup).
3. After copying the database, A, B and C now had the actual restored read status, D and E were no longer being shown.
4. I copied D and E back from Calibre to the ereader so that they would show up again, but then it'd optimize and also reset the status for A, B and C.

I've also just tried your other suggestion. I copied the sqlite database back to the device, it now shows the read progress on the ereader, and all the annotations.
However when I now try to store the bookmarks it says:

calibre, version 7.17.0
Kobo Utilities - Library updated: Update summary:
Reading locations updated=0
Books with no reading location=1
Total books checked=1

As if it just doesn't really fully recognize the database that was copied to it. At the same time, Calibre does show a checkmark before the books it finds.

[Xylon]

And one more thing I just noticed: I copied the backup to the device, rebooted it, it shows the read status and annotations on all books. But when I then connect it to the pc, open calibre, tell calibre to eject the device, it'll optimize the books again and drop all the info it has like the read status and resets everything to Unread.

So I don't even need to actually copy books for it to reset everything after it's been connected to Calibre.

[JSWolf]

Originally Posted by Xylon

Interesting, thank you.

So to make the initial example a bit clearer:

1. my case for example I've copied books A, B, C, D and E to the ereader. All stats were set to unread automatically.
2. I then copied the sqlite file, which had information about A, B and C (D and E were added later and were not in this backup).
3. After copying the database, A, B and C now had the actual restored read status, D and E were no longer being shown.
4. I copied D and E back from Calibre to the ereader so that they would show up again, but then it'd optimize and also reset the status for A, B and C.

I've also just tried your other suggestion. I copied the sqlite database back to the device, it now shows the read progress on the ereader, and all the annotations.
However when I now try to store the bookmarks it says:

calibre, version 7.17.0
Kobo Utilities - Library updated: Update summary:
Reading locations updated=0
Books with no reading location=1
Total books checked=1

As if it just doesn't really fully recognize the database that was copied to it. At the same time, Calibre does show a checkmark before the books it finds.

Did you reboot after ejecting the Kobo from the compter?

[DNSB]

1. connected and copied over the databases, safely ejected and restarted your Kobo. This needs to be the first step so you have a good database for calibre to work with.
2. copied books A, B, C, D & E to your Kobo
3. safely ejected and let the books be imported
4. Reconnect your Kobo to calibre
5. Select all the books shown as being on your device in calibre
6. In the Kobo Utilities menu, select update metadata and click on OK.
7. In the Kobo Utilities menu, select Store/restore current bookmark. Make sure the Restore radio button is selected and click on OK.
8. Safely eject your ereader

As for the bookmark restore, did you configure the needed custom columns and Kobo Utilities is configured to use them so you have some data to restore? See 1st attached image. the 4th image is what the columns look like in my calibre library view.

[Xylon]

Originally Posted by JSWolf

Did you reboot after ejecting the Kobo from the compter?

I did. What I did so far was use explorer to eject it, wait a bit, and then turn it on/off while it's still connected. So at that point it was still showing the black "charging" screen.

This time I did it a bit differently. I connected it, copied the sqlite files, ejected it through explorer, and then pulled out the cable from the ereader. For the first few seconds I could see the status of the books and the progress. However then it started importing content progress bar and I could see the books resetting to unread one by one.

Originally Posted by DNSB

1. connected and copied over the databases, safely ejected and restarted your Kobo. This needs to be the first step so you have a good database for calibre to work with.
2. copied books A, B, C, D & E to your Kobo
3. safely ejected and let the books be imported
4. Reconnect your Kobo to calibre
5. Select all the books shown as being on your device in calibre
6. In the Kobo Utilities menu, select update metadata and click on OK.
7. In the Kobo Utilities menu, select Store/restore current bookmark. Make sure the Restore radio button is selected and click on OK.
8. Safely eject your ereader

As for the bookmark restore, did you configure the needed custom columns and Kobo Utilities is configured to use them so you have some data to restore? See 1st attached image. the 4th image is what the columns look like in my calibre library view.

I didn't have the columns yet, just created those. I'll give it a go and get back in a bit. But all of this takes a LONG time, because somehow I have to keep rebooting the pc or the ereader every time after disconnecting it, because otherwise it doesn't recognize it and the ereader just reboots.

[Xylon]

Ok, what I've done now:

1. Factory reset
2. Set up the ereader (log in)
3. Connect to PC with calibre and stuff closed
4. Rename BookReader and KoboReader files on ereader and copy the two files from the backup
5. Pressed Eject in the toolbar in windows
6. Screen kept showing "connected and charging" after eject, but Windows was no longer showing it as connected, so I pulled out the cable
7. It briefly showed a list of books with reader percentage and the name as cover art cause the files didn't exist, but after a second they started disappearing one by one
8. Turned it off and back on again, no books in list
9. Connect it to PC again
10. Open Calibre, this now shows no books as being present
11. Send all by books to the device
12. Eject it and the ereader imported everything. Everything shows up as unread
13. Reconnect to PC and calibre
14. Selected all the books with green checkmark
15. Run update metadata with same checkboxes as in screenshot #2 (shows 14 books updated)
16. Go to store/restore current bookmark and run restore with settings in #3
17. Eject device
18. Everything still shows as unread

I think the restore bookmarks didn't do a lot as I just created the columns earlier today and they weren't filled. That's at least what I was hoping to use the backed up sqlite files for.

I feel like the ereader simply isn't accepting the changed database, and as soon as it sees is it tries to "update" it which boils down to resetting it.

[JSWolf]

Originally Posted by Xylon

I did. What I did so far was use explorer to eject it, wait a bit, and then turn it on/off while it's still connected. So at that point it was still showing the black "charging" screen.

You have to wait until you are back at the home screen before you reboot.

Also, you should delete any eBooks on your Kobo that are not in the backup database. You want to make sure that none of the corrupted database is in memory when the eBooks are imported.

[Xylon]

Originally Posted by JSWolf

You have to wait until you are back at the home screen before you reboot..

How long does that usually take for you when ejecting from explorer? Ejecting with Calibre or Kobo has it go back to the home screen almost immediately. When I eject with explorer I have no idea if/when that happens. I've waited a few minutes already once, but no luck, unfortunately.

Originally Posted by JSWolf

Also, you should delete any eBooks on your Kobo that are not in the backup database. You want to make sure that none of the corrupted database is in memory when the eBooks are imported

Ooh good point. That seems to have been solved by the factory resets at least. Both times it was completely empty.

[Xylon]

Originally Posted by JSWolf

You have to wait until you are back at the home screen before you reboot.

Also, you should delete any eBooks on your Kobo that are not in the backup database. You want to make sure that none of the corrupted database is in memory when the eBooks are imported.

Ok, new try:

1. Factory reset
2. Device is empty except for default files
3. Remove the old sqlite files (let's try removing them instead of just renaming them)
4. Copy the backed up sqlite files to the device
5. Open kobo app
6. Eject through kobo
7. It shows the message "You've ejected your ereader. You can now unplug it".
8. Five minutes later it's still showing the message

Unfortunately even ejecting through Kobo doesn't appear to remove the black screen on the ereader. And removing the cable briefly shows the books again, before they disappear one by one. I'll give it one more try and see if ejecting it through Calibre next time works.

[Xylon]

Gave it two more tries.

First try was the same as above, only ejecting through Calibre. It briefly showed the books, before they disappeared one by one.

Second try as the one above up and including step 4, only then:

5. Open Calibre
6. Send books to device
7. Eject
8. Importing books

So far it seems to keep the books and status. I don't have any more time, but I'll add some more books later and see if it resets everything again.

[DNSB]

Originally Posted by Xylon

How long does that usually take for you when ejecting from explorer? Ejecting with Calibre or Kobo has it go back to the home screen almost immediately. When I eject with explorer I have no idea if/when that happens. I've waited a few minutes already once, but no luck, unfortunately.

Are you ejecting by right clicking on the Kobo drive in File Explorer and then clicking on eject or by using the USB widget in the systray. I find that ejecting from File Explorer works (power is removed from the USB port for a short while) while ejecting from the USB widget does not work (either power is not dropped or it is dropped for a shorter time than my cheap USB gadget can catch).

[Xylon]

Originally Posted by DNSB

Are you ejecting by right clicking on the Kobo drive in File Explorer and then clicking on eject or by using the USB widget in the systray. I find that ejecting from File Explorer works (power is removed from the USB port for a short while) while ejecting from the USB widget does not work (either power is not dropped or it is dropped for a shorter time than my cheap USB gadget can catch).

Aaaah yeah, I was using the one in the task bar in the bottom right. I was assuming both would call the same method to eject. Good catch. The one through the actual file explorer, right next to the other disks, does work perfectly fine.

But I just pushed a new ebook to the ereader to trigger the optimization step again to see what happens, and it looks like everything is still fine! One or two are set to unread, but everything that was in progress still is in progress I think.

So looks like what worked for me at least was replacing the sqlite db and pushing the books during the same connected session:

1. Factory reset (Device is empty except for default files)
2. Remove/rename the old sqlite files
3. Copy the backed up sqlite files to the device
4. Open Calibre
5. Send books to device
6. Eject device

Thanks both of you for the support and advice!

[DNSB]

Good to hear that your Kobo is a happy camper again.