[Kobo] Database corruption - thoughts?

jonmach · 11-09-2023, 06:37 AM

I've had repeated corruption issues on my Kobo for the last few months. I even got my Kobo (Sage) replaced. It has the latest firmware. The new one has the same issues with database corruption. I can only assume that the problem is with a recent Calibre update.

Spoiler:

kovidgoyal · 11-09-2023, 07:54 AM

There have been no changes to the kobo driver in calibre in years, indeed the person that used to maintain the driver passed away so it is in pure maintenance mode. But feel free to downgrade calibre to whatever version you like. There is an entry in the calibre FAQ about the kobo database corruption problem, read it.

JSWolf · 11-09-2023, 09:03 AM

Plug in the Kobo. Copy the last backed-up database over the existing database. Eject from the computer and reboot the Kobo. That should do it.

What firmware are you using on your Kobo? What I do is I make sure I go back to the home screen before I plug my Libra 2 into the computer. Some others do a reboot before plugging into the computer.

theducks · 11-09-2023, 11:02 AM

Ejecting from Calibre does not mean the transfer is fully complete either.
I wait for the Processing screen (if a new book is added) or the home screen to be stable (a few seconds)

DNSB · 11-09-2023, 02:24 PM

I ran into the database corruption a while back and my workaround is to reboot my Kobo Sage before connecting to my computer. I use a menu item in NickelMenu to do this. I can do the second connection to send the metadata for newly added books without needing a reboot but any time I've been reading, the reboot keeps me from seeing database corruption.

You may want to take a look at the Firmware 4.38.21908 thread in the Kobo forum for more discussion.

Kobo is aware of the issue and it should to be fixed in the next release.

ownedbycats · 11-10-2023, 01:19 PM

The current Kobo firmware has a glitch that sometimes leaves things open when connecting to PC. That makes database corruption more likely. (If you look at the .kobo folder in your file manager, you may see an sqlite write-ahead log - which indicates that it exported to USB while the sqlite database was open.)

I've had it happen under two circumstances:

a) connecting while a kepub is open.
b) Using a NickelMenu option to reconnect immediately after importing a large amount of books.

Most of the time, making sure I'm on home screen before connecting prevents it. However, I do reboot after doing imports.

Quote:

Originally Posted by JSWolf

Plug in the Kobo. Copy the last backed-up database over the existing database. Eject from the computer and reboot the Kobo. That should do it.

If you have not a backup, 'repair account' in the Kobo options I think will also fix it.

DNSB · 11-10-2023, 04:59 PM

Quote:

Originally Posted by ownedbycats

If you have not a backup, 'repair account' in the Kobo options I think will also fix it.

Generally, I use signout/signin to restore the database to a usable condition but you will lose status, annotations, etc. The restore from backup minimizes the losses so it's my preferred option.

ownedbycats · 11-10-2023, 10:08 PM

Quote:

Originally Posted by DNSB

Generally, I use signout/signin to restore the database to a usable condition but you will lose status, annotations, etc. The restore from backup minimizes the losses so it's my preferred option.

Yeah, restoring backup is best option.

I've used Repair Account the first time I got a database corruption - then immediately set up Kobo Utilities to make daily backups. Is possible that it doesn't work in all cases though.

theducks · 11-10-2023, 11:27 PM

Since I have 2 Kobo's, I made folders for each (and the settings in utilities)
I also bumped the number of copies to 10 since I might connect a number of times while fine tuning a book

Jasmine GreenTea · 11-24-2023, 06:52 AM

Hallo! I too have run into a database image corruption. Calibre cannot connect to my Kobo Elipsa 2E because of this "malformed database image." The last added books (purchased and sideloaded) have become invisible.

I have tried Repair from the Kobo Settings menu, but to no avail.

I understand that Logging out from my Kobo account, and then Logging in could help reinitialize a database. Is this in any way different from a Factory rested?

In May I went through a Factory reset with my Aura H2O e2, and that was... traumatic. I tend to make a lot of annotations and notes, and almost of those were lost. However, as I had a full manual backup of the content of that ereader, I was able to recover those annotations.

At this moment, on my computer I do have a full copy of everything that was on my Elipsa (but I do not have a copy of the database taken with KoboUtilities). Once I copy everything back on the Elipsa, will the annotations be available again? And what about the handwritten stylus notes?

Quoth · 11-24-2023, 08:57 AM

I always export notes. Then there is a folder with them that work (text, docx, html, png, pdf). Mostly I export docx from Advanced and png from basic.
Never use pen for epubs, use text as the pen annotations are useless on PC. I think the pen annotations make extra content on a PDF, so you can copy back the pdf. I always read annotations when I finish an epub using Kobo Utilities & copy/paste to a text file.

You can copy the native folders / notes and when copied back after a factory reset, they are re-imported to the database. Probably only the metadata such as names and kind of the notes (basic/advanced) is in the database as I've copied the actual notebook files/foders from one kobo to another and they appear.

A Factory reset will reset & erase most everything.

A logout / login will build a fresh database, but may erase notes folders you created (they can be backed up by file copy). It may erase annotations/highlights/bookmarks.

DNSB · 11-26-2023, 01:12 AM

I would also recommend installing NickelMenu and rebooting before triggering a USB connect to your computer. This ensures the database is closed before the connection.

Jasmine GreenTea · 11-27-2023, 06:29 AM

Thank you for your prompl and informative reply.

I am not concerned about the Handwirtten nor trhe OCR-ed notebooks made on Elipsa, as I have copies of those on DropBox and also on my computer.

I am more concerned about the highlights and occasional annotations to EPUBs.

Quote:

Originally Posted by Quoth

A logout / login will build a fresh database, but may erase notes folders you created (they can be backed up by file copy). It may erase annotations/highlights/bookmarks.

So, I understand there's no way to know which of these would be gone after a logout/login.

As I do have a file-by-file backup of the content of the ereader, would that help? Would it help in case I find that the logout/login erases too much?

At this moment, I am highlighting manually the most important passages on another ereader. But, man, this is boring and time-consuming!

Quoth · 11-27-2023, 08:59 AM

Quote:

Originally Posted by Jasmine GreenTea

I am more concerned about the highlights and occasional annotations to EPUBs.

Export these using Kobo Utilities, except that has to access the database.

You can add a config menu item to export annotations per book on the Kobo, but if the database is damaged it may not work.

There is though currently no way to import other than restoring a backed up database. Except your database is corrupt.

Even if you could re-import annotations, the ebook needs to be same name and identical and in same database record for a separate import of saved annotations to work (which isn't a thing that exists. You can maybe restore bookmarks, reading position and reading status using Kobo Utilities, which can make a backup of that. Not sure why it doesn't also do highlights and annotations. But, again, the database needs to be sufficiently intact.

Quote:

As I do have a file-by-file backup of the content of the ereader, would that help? Would it help in case I find that the logout/login erases too much?

The files have the ebooks, notebooks and settings. Annotations & highlights and bookmarks are in the sqlight database file. The logout & login deletes the database and makes a new one, and any folders/directories created in the Notebook section (and thus their contents, but that can likely be backed up and restored, though I only tried actual notebooks, not folders created via recent FW GUI). Actual ebooks and other notebooks are not deleted at all, but the annotations are in the database file, not the actual ebooks, which makes sense actually.

Jasmine GreenTea · 11-28-2023, 03:01 PM

@Quoth -- Thank you, this is really helpful.

I see that the way highlights and annotations are recorded on my Elipsa is different from the one on Aura H2O. If the former keeps that data in the SQL database, whereas the Aura stores it in a folder called Annotations (which actually refers also to Highlights), where each author has their own folder that includes a file with the extension .annot which contains the text of each annotation in clear text, set between paired tags like this:

Quote:

<annotation>
<dc:identifier>urn:uuid:808d764c-7644-4f57-9637-6e24966182c6</dc:identifier>
<dc:date>2018-05-15T18:00:07Z</dc:date>
<target>
<fragment start="OEBPS/Text/index_split_017.html#point(/1/4/288/3:39)" end="OEBPS/Text/index_split_017.html#point(/1/4/290/1:15)" progress="0">
<text>the sighting of the mainland – kinara agil hai! – was the spell broken. The lookout’s shout set off a race to the maindeck; even some of the wounded, barely able to stand on their own feet, went hobbling forward to catch their first glimpse of the land of Maha-chin.</text>
</fragment>
</target>
<content>
<dc:date>2018-05-15T18:00:07Z</dc:date>
</content>
</annotation>

When I had to reset my Aura, I was able recover my highlights and annotations by simply copying these files from my computer back to the ereader. (These were all third-party books).

I wonder why did Kobo abandon this system, which was more user friendly.

11-09-2023, 06:37 AM	#1
jonmach Junior Member Posts: 1 Karma: 10 Join Date: Sep 2023 Device: Kobo	[Kobo] Database corruption - thoughts? I've had repeated corruption issues on my Kobo for the last few months. I even got my Kobo (Sage) replaced. It has the latest firmware. The new one has the same issues with database corruption. I can only assume that the problem is with a recent Calibre update. Spoiler: calibre, version 6.29.0 ERROR: Error: Error communicating with device CorruptError: database disk image is malformed Traceback (most recent call last): File "calibre/gui2/device.py", line 104, in run File "calibre/gui2/device.py", line 558, in _books File "calibre/devices/kobo/driver.py", line 2028, in books File "src/cursor.c", line 240, in resetcursor apsw.CorruptError: CorruptError: database disk image is malformed I can't think of anything I'm doing that would cause it. I plug the device in, and the Mac recognises it fine. I start Calibre, and it throws an error. I had previously backed the DB up using KoboUtilies, but I can't see how to restore a previous database. The cable connection is fine. I even used a new Kobo cable as well, so I doubt this is it. Any ideas? The alternative is that I keep factory resetting the device. I've lost count of how many times I've done this. Reset, resync with Kobo app etc etc. As I said, this is a new thing. I've used Calibre with a Kobo for years with zero issues. I have also removed the /Volumes/KOBOReader from the 'Microsoft Defender' scan list. The database does indeed seem to be corrupt. A Database check shows the following Result of running 'PRAGMA integrity_check' on database on the Kobo device: * in database main * Tree 47 page 136545 cell 0: invalid page number 218103808 Tree 3 page 136674 cell 32: 2nd reference to page 132808 Tree 3 page 41277 cell 16: Rowid 117479 out of order Tree 3 page 24198 cell 41: Rowid 109702 out of order Tree 3 page 132783: btreeInitPage() returns error code 11 Tree 3 page 24198 cell 32: 2nd reference to page 132776 Tree 3 page 7224 cell 0: Rowid 60655 out of order Tree 3 page 4951 cell 82: 2nd reference to page 132814 Tree 3 page 4951 cell 79: Rowid 99066 out of order Tree 3 page 4951 cell 75: 2nd reference to page 132811 Tree 3 page 4951 cell 58: Rowid 92758 out of order Tree 3 page 4951 cell 53: Rowid 92129 out of order Tree 3 page 4951 cell 51: Rowid 76756 out of order Tree 3 page 4951 cell 49: 2nd reference to page 132761 Tree 3 page 4951 cell 44: 2nd reference to page 132764 Tree 3 page 4951 cell 41: 2nd reference to page 132760 Tree 3 page 4779 cell 3: Rowid 109191 out of order Tree 3 page 132740: btreeInitPage() returns error code 11 Tree 3 page 132007 cell 0: Rowid 59171 out of order Tree 3 page 98295 cell 2: 2nd reference to page 132836 Tree 3 page 98295 cell 1: invalid page number 218103808 Tree 3 page 98295 cell 0: 2nd reference to page 132817 Tree 3 page 81117 cell 2: invalid page number 218103808 Tree 3 page 81117 cell 1: 2nd reference to page 132809 Tree 3 page 81117 cell 0: invalid page number 218103808 Tree 3 page 67803 cell 0: invalid page number 33554432 Tree 3 page 45770 cell 1: overflow list length is 3 but should be 4 Tree 3 page 45770 cell 0: invalid page number 33554432 Tree 3 page 42764 cell 1: invalid page number 167772160 Tree 3 page 42764 cell 0: overflow list length is 1 but should be 2 Tree 3 page 18049 cell 0: invalid page number 167868928 Tree 3 page 11582 cell 1: 2nd reference to page 132765 Tree 3 page 11582 cell 0: 2nd reference to page 132762 Tree 3 page 9288 cell 0: overflow list length is 4 but should be 5 Tree 3 page 9193 cell 0: overflow list length is 3 but should be 6 Tree 3 page 9190 cell 0: 2nd reference to page 132750 Tree 3 page 4951 cell 1: Child page depth differs Tree 3 page 4951 cell 0: Child page depth differs Tree 3 page 5882 cell 0: 2nd reference to page 132812 Tree 3 page 4619 cell 94: Child page depth differs Tree 3 page 4984 cell 0: invalid page number 33628160 Tree 3 page 4619 cell 93: Child page depth differs Tree 80 page 7024 cell 5: 2nd reference to page 132742 Tree 80 page 7024 cell 5: Child page depth differs Tree 80 page 7024 cell 4: Child page depth differs Tree 80 page 18147 cell 0: 2nd reference to page 132741 Tree 80 page 18147 cell 0: Child page depth differs Tree 80 page 7024 cell 2: Child page depth differs Tree 80 page 7024 cell 1: Child page depth differs Tree 80 page 136614 cell 5: 2nd reference to page 132813 Tree 4 page 9675 cell 2: 2nd reference to page 132787 Tree 4 page 9675 cell 3: Child page depth differs Tree 4 page 5652 cell 2: 2nd reference to page 132784 Tree 11 page 132939 cell 29: 2nd reference to page 132835 Tree 11 page 8226 cell 50: 2nd reference to page 132820 Tree 11 page 8226 cell 12: 2nd reference to page 132798 Tree 11 page 8226 cell 11: 2nd reference to page 132799 Tree 11 page 133474 cell 52: 2nd reference to page 132746 Tree 11 page 133474 cell 40: 2nd reference to page 132739 Tree 11 page 6045 cell 0: Rowid 113879 out of order Tree 11 page 58702 cell 6: 2nd reference to page 132815 Tree 11 page 9620 cell 57: Rowid 76675 out of order Tree 11 page 9620 cell 55: 2nd reference to page 132777 Tree 11 page 5257 cell 1: Rowid 71872 out of order Tree 11 page 9620 cell 47: 2nd reference to page 132759 Tree 11 page 9620 cell 44: Rowid 65133 out of order Tree 11 page 9620 cell 43: 2nd reference to page 132758 Tree 11 page 9620 cell 39: Rowid 59368 out of order Tree 81 page 136618 cell 4: 2nd reference to page 132767 Tree 12 page 91078 cell 0: 2nd reference to page 132749 Tree 12 page 91078 cell 3: Child page depth differs Tree 12 page 9421 cell 3: 2nd reference to page 132792 Tree 12 page 11819 cell 0: 2nd reference to page 132748 Tree 12 page 11819 cell 3: Child page depth differs Tree 12 page 5313 cell 2: 2nd reference to page 132794 Tree 12 page 5313 cell 4: 2nd reference to page 132747 Tree 12 page 136591 cell 3: 2nd reference to page 132778 Tree 12 page 136591 cell 1: Child page depth differs Tree 12 page 5665 cell 0: 2nd reference to page 132783 Tree 12 page 5612 cell 8: 2nd reference to page 132779 Tree 12 page 5313 cell 3: Child page depth differs Tree 12 page 136630 cell 2: 2nd reference to page 132781 Tree 12 page 5477 cell 3: 2nd reference to page 132782 Tree 12 page 4970 cell 6: 2nd reference to page 132752 Tree 12 page 4906 cell 1: 2nd reference to page 132756 Tree 12 page 998 cell 3: 2nd reference to page 132740 Page 5804: never used Page 5807: never used Page 5808: never used Page 5813: never used Page 5877: never used Page 6008: never used Page 6028: never used Page 6088: never used Page 6091: never used Page 6101: never used Page 6110: never used Page 6117: never used Page 6119: never used Page 6121: never used Last edited by theducks; 11-09-2023 at 10:59 AM. Reason: spoilered log

11-24-2023, 06:52 AM	#10
Jasmine GreenTea Enthusiast Posts: 28 Karma: 10 Join Date: Oct 2012 Device: Kobo Elipsa E2, Kobo Aura H20 ed 2, Kobo Clara, Sony PRS T2 & T3	Database malformed on Elipsa 2E Hallo! I too have run into a database image corruption. Calibre cannot connect to my Kobo Elipsa 2E because of this "malformed database image." The last added books (purchased and sideloaded) have become invisible. I have tried Repair from the Kobo Settings menu, but to no avail. I understand that Logging out from my Kobo account, and then Logging in could help reinitialize a database. Is this in any way different from a Factory rested? In May I went through a Factory reset with my Aura H2O e2, and that was... traumatic. I tend to make a lot of annotations and notes, and almost of those were lost. However, as I had a full manual backup of the content of that ereader, I was able to recover those annotations. At this moment, on my computer I do have a full copy of everything that was on my Elipsa (but I do not have a copy of the database taken with KoboUtilities). Once I copy everything back on the Elipsa, will the annotations be available again? And what about the handwritten stylus notes? Last edited by Jasmine GreenTea; 11-24-2023 at 08:08 AM.

11-24-2023, 08:57 AM	#11
Quoth the rook, bossing Never. Posts: 12,342 Karma: 92073397 Join Date: Jun 2017 Location: Ireland Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11	I always export notes. Then there is a folder with them that work (text, docx, html, png, pdf). Mostly I export docx from Advanced and png from basic. Never use pen for epubs, use text as the pen annotations are useless on PC. I think the pen annotations make extra content on a PDF, so you can copy back the pdf. I always read annotations when I finish an epub using Kobo Utilities & copy/paste to a text file. You can copy the native folders / notes and when copied back after a factory reset, they are re-imported to the database. Probably only the metadata such as names and kind of the notes (basic/advanced) is in the database as I've copied the actual notebook files/foders from one kobo to another and they appear. A Factory reset will reset & erase most everything. A logout / login will build a fresh database, but may erase notes folders you created (they can be backed up by file copy). It may erase annotations/highlights/bookmarks. Last edited by Quoth; 11-24-2023 at 09:01 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Database Corruption / FW 3.5.0	PeterT	Kobo Reader	7	08-26-2014 05:10 PM
Database Corruption during Installation	aronan	Calibre	3	09-10-2010 02:17 AM
Problems after database corruption	ficbot	Calibre	2	09-25-2009 03:11 PM
Database Corruption!!	Alexander Turcic	Announcements	0	07-22-2003 11:08 AM

11-09-2023, 07:54 AM	#2
kovidgoyal creator of calibre Posts: 44,526 Karma: 24495948 Join Date: Oct 2006 Location: Mumbai, India Device: Various	There have been no changes to the kobo driver in calibre in years, indeed the person that used to maintain the driver passed away so it is in pure maintenance mode. But feel free to downgrade calibre to whatever version you like. There is an entry in the calibre FAQ about the kobo database corruption problem, read it.

11-09-2023, 09:03 AM	#3
JSWolf Resident Curmudgeon Posts: 76,370 Karma: 136466962 Join Date: Nov 2006 Location: Roslindale, Massachusetts Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3	Plug in the Kobo. Copy the last backed-up database over the existing database. Eject from the computer and reboot the Kobo. That should do it. What firmware are you using on your Kobo? What I do is I make sure I go back to the home screen before I plug my Libra 2 into the computer. Some others do a reboot before plugging into the computer.

11-09-2023, 11:02 AM	#4
theducks Well trained by Cats Posts: 30,442 Karma: 58055868 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	Ejecting from Calibre does not mean the transfer is fully complete either. I wait for the Processing screen (if a new book is added) or the home screen to be stable (a few seconds)

11-09-2023, 02:24 PM	#5
DNSB Bibliophagist Posts: 40,487 Karma: 156982136 Join Date: Jul 2010 Location: Vancouver Device: Kobo Sage, Libra Colour, Lenovo M8 FHD, Paperwhite 4, Tolino epos	I ran into the database corruption a while back and my workaround is to reboot my Kobo Sage before connecting to my computer. I use a menu item in NickelMenu to do this. I can do the second connection to send the metadata for newly added books without needing a reboot but any time I've been reading, the reboot keeps me from seeing database corruption. You may want to take a look at the Firmware 4.38.21908 thread in the Kobo forum for more discussion. Kobo is aware of the issue and it should to be fixed in the next release.

11-10-2023, 11:27 PM	#9
theducks Well trained by Cats Posts: 30,442 Karma: 58055868 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	Since I have 2 Kobo's, I made folders for each (and the settings in utilities) I also bumped the number of copies to 10 since I might connect a number of times while fine tuning a book

11-26-2023, 01:12 AM	#12
DNSB Bibliophagist Posts: 40,487 Karma: 156982136 Join Date: Jul 2010 Location: Vancouver Device: Kobo Sage, Libra Colour, Lenovo M8 FHD, Paperwhite 4, Tolino epos	I would also recommend installing NickelMenu and rebooting before triggering a USB connect to your computer. This ensures the database is closed before the connection.

Advert

Advert