Unable to restore databasecalibre

asw22 · Today, 09:39 AM

Hello everyone!

An thanks to Kovid for this wonderful too that is Calibre!

I have been a Calibre user for about ten years (now 7.15/W11), and I manage about 45000 books, without too many issues...However the issues I have had recently are a bit strange:

1) on my main library, when I make a research via author or series, some books are missing for no reason...and yet they are in the library folder in their proper place, and so far (say one year ago) there was no issue...
So I tried to restore the database...same issue!

2) I made an experiment and duplicated the Calibre folder under another name via the export/import process using an intermediary folder...I obtain an exact duplicate of the original database, with the same books still not considered in the metadata.db

3) Having a full library to play with, I try to restore the database: this time the original database (75M) is "destroyed" (albeit saved) and replaced by a 5 book database or so (404K)!!! I try again, same thing!

4) I try a last test, and remove the metadata.db, hoping the restauration will force Calibre to recreate it via the books data in the Calibre folder (like it is supposed to do), thus allowing me to get all the books: no dice I still get the same small database!!!

What could happen?

Thanks for your lights!

asw22

kovidgoyal · Today, 10:42 AM

Use the check library tool to find all such problems and then you can re-add the books. Or if you just want to do it in bulk you can create a new empty library then use the add books -> add from folders -> one book per folder to recreate the library from your old library folder.

asw22 · Today, 11:28 AM

Thanks Kovid!

I am trying the first method but the results window (which shows all the types of issue!) is blocked (not responding). I guess this is due to the very large size of the library and I just need to be patient..

I will try the second method later when Calibre will have given me control on operations...

Sirtel · Today, 11:38 AM

You should also try to find out what has caused your database issues, to prevent them from happening again. Is your library on a cloud or a network drive? Have you added/deleted books directly from the library folder, without doing it via the calibre GUI? Those are the most common reasons for data corruption.

asw22 · Today, 12:04 PM

I never added books in another way than via Calibre.

I did however get many duplicates (for various reasons, but often because Calibre did not show me all the books!), and I removed some via Calibre and others directly in the library, but always by removing the containing folder of the book.

For unfathomable reasons, I got myself many books without any format (about 10000 out of 50000) I am dumbfounded, to say the least!

Sirtel · Today, 12:27 PM

Quote:

Originally Posted by asw22

...and others directly in the library, but always by removing the containing folder of the book.

Never do that. The book entry remains in the database that way.

Any chance of cloud syncing? OneDrive, Google Drive, an automatic backup solution?

theducks · Today, 12:36 PM

Quote:

Originally Posted by asw22

I never added books in another way than via Calibre.

I did however get many duplicates (for various reasons, but often because Calibre did not show me all the books!), and I removed some via Calibre and others directly in the library, but always by removing the containing folder of the book.

For unfathomable reasons, I got myself many books without any format (about 10000 out of 50000) I am dumbfounded, to say the least!

2-way SYNC software is the most likely cause.
You fix a title or Author and the sync puts the old one back. OR you use Google Drive (which damages the file name ) Do read the FAQ about NAS shares.

asw22 · Today, 01:14 PM

All what I have described is happening on the same NVE SSD (and other internal disks in the same computer). I don't use cloud or NAS.

I have done what Kovid suggested (create an empty library and filling it with the books of another one): this work fine except I los all the metadata painstakingly assembled/created from different sources (series, blurb...): this technique allows retrieval of the epub, but not the opf!
How can I fix that? I would have 45000 sets of metadata to retrieve and adjust...

PS I am playing with copies, the original library is intact!

JSWolf · Today, 01:26 PM

Quote:

Originally Posted by asw22

All what I have described is happening on the same NVE SSD (and other internal disks in the same computer). I don't use cloud or NAS.

I have done what Kovid suggested (create an empty library and filling it with the books of another one): this work fine except I los all the metadata painstakingly assembled/created from different sources (series, blurb...): this technique allows retrieval of the epub, but not the opf!
How can I fix that? I would have 45000 sets of metadata to retrieve and adjust...

PS I am playing with copies, the original library is intact!

Select the original library. In the toolbar lick on the library icon. Then select Restore Database. That will build a new database from your eBooks in the library. I'm not sure if the option to check the library comes up after that. If it does, do it. If not, select Library maintenance and check the library.

asw22 · Today, 04:40 PM

I did that, and the original library database was "properly" restored, with the same issues as before:

1) I have a list of a dozen books without format...for instance I have a series of 14, for which there are 14 folders in the library including two without format: only two books shown in Calibre, and the two without format also shown?? It totally baffles me!

2) another series of 12, all ok except number 2, which is properly present in the library with its 11 other issues, except this one is not shown by Calibre!

I have no idea what happens...

3) I have many duplicate folders, one complete, one with just the opf. The verification of library seems to identify them, but I seem unable to use the verification process properly: even after 1/2h waiting the process is still non responsive...

I wonder if a interesting way to recreate a library would be to use an improvement to the technique mentioned by Kovid of extracting only books from the source library (which is the case now with the loss of all cover & metadata), but the individual book folders, including cover and metadata, with checking there is a real book inside? Is there already a way to do that?

Thanks to all of you helping me!

asw22

Sirtel · Today, 04:51 PM

It seems something has thoroughly messed up your library. What you're seeing is in no way normal. Are you sure you don't have any syncing or backing-up program running in the background?

Of course, if you've been deleting books from your library folder without going through the Calibre GUI, that would also mess things up. But I think not as badly as you described.

JSWolf · Today, 04:58 PM

Quote:

Originally Posted by Sirtel

It seems something has thoroughly messed up your library. What you're seeing is in no way normal. Are you sure you don't have any syncing or backing-up program running in the background?

Of course, if you've been deleting books from your library folder without going through the Calibre GUI, that would also mess things up. But I think not as badly as you described.

Something has to be doing this. It doesn't just happen.

s @asw22 going into the library and changing things? I agree that it could be some sort of sync program maybe syncing to/from Google Drive or some sort of corrupt library. I don't know what gets corrupted from Google Drive. I just know it does get corrupted.

I can't see it being an AV program causing the problem. But if it is, that AV program would have to go.

The thing to do (IMHO) is to copy the existing library and clean up the library and get it as good as it can be and then do a proper backup. But before doing that, put the library someplace else so if it is a sync program, it won't sync the new library.

Sirtel · Today, 05:57 PM

Quote:

Originally Posted by JSWolf

Something has to be doing this. It doesn't just happen.

s @asw22 going into the library and changing things? I agree that it could be some sort of sync program maybe syncing to/from Google Drive or some sort of corrupt library. I don't know what gets corrupted from Google Drive. I just know it does get corrupted.

It doesn't need to be Google Drive. Any sync program that runs in the background all the time could do this.

Today, 09:39 AM	#1
asw22 Junior Member Posts: 5 Karma: 10 Join Date: Aug 2024 Device: PocketBook InkPad 3	Unable to restore databasecalibre Hello everyone! An thanks to Kovid for this wonderful too that is Calibre! I have been a Calibre user for about ten years (now 7.15/W11), and I manage about 45000 books, without too many issues...However the issues I have had recently are a bit strange: 1) on my main library, when I make a research via author or series, some books are missing for no reason...and yet they are in the library folder in their proper place, and so far (say one year ago) there was no issue... So I tried to restore the database...same issue! 2) I made an experiment and duplicated the Calibre folder under another name via the export/import process using an intermediary folder...I obtain an exact duplicate of the original database, with the same books still not considered in the metadata.db 3) Having a full library to play with, I try to restore the database: this time the original database (75M) is "destroyed" (albeit saved) and replaced by a 5 book database or so (404K)!!! I try again, same thing! 4) I try a last test, and remove the metadata.db, hoping the restauration will force Calibre to recreate it via the books data in the Calibre folder (like it is supposed to do), thus allowing me to get all the books: no dice I still get the same small database!!! What could happen? Thanks for your lights! asw22

Today, 11:38 AM	#4
Sirtel Grand Sorcerer Posts: 10,781 Karma: 226728226 Join Date: Jan 2014 Location: Estonia Device: Kobo Sage & Libra 2	You should also try to find out what has caused your database issues, to prevent them from happening again. Is your library on a cloud or a network drive? Have you added/deleted books directly from the library folder, without doing it via the calibre GUI? Those are the most common reasons for data corruption. Last edited by Sirtel; Today at 12:30 PM. Reason: typo

Today, 04:40 PM	#10
asw22 Junior Member Posts: 5 Karma: 10 Join Date: Aug 2024 Device: PocketBook InkPad 3	I did that, and the original library database was "properly" restored, with the same issues as before: 1) I have a list of a dozen books without format...for instance I have a series of 14, for which there are 14 folders in the library including two without format: only two books shown in Calibre, and the two without format also shown?? It totally baffles me! 2) another series of 12, all ok except number 2, which is properly present in the library with its 11 other issues, except this one is not shown by Calibre! I have no idea what happens... 3) I have many duplicate folders, one complete, one with just the opf. The verification of library seems to identify them, but I seem unable to use the verification process properly: even after 1/2h waiting the process is still non responsive... I wonder if a interesting way to recreate a library would be to use an improvement to the technique mentioned by Kovid of extracting only books from the source library (which is the case now with the loss of all cover & metadata), but the individual book folders, including cover and metadata, with checking there is a real book inside? Is there already a way to do that? Thanks to all of you helping me! asw22 Last edited by asw22; Today at 04:43 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Unable to restore backup from Google drive in Moonreader pro due to blocked access	archz2	Android Devices	0	05-12-2024 11:32 AM
iPhone 7 unable to restore Error 4013	Dean11	Apple Devices	1	10-23-2018 01:09 PM
version 3.15.0: unable to 'restore' database	BuPunkt	Library Management	4	01-18-2018 08:45 AM

Today, 10:42 AM	#2
kovidgoyal creator of calibre Posts: 44,169 Karma: 22670164 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Use the check library tool to find all such problems and then you can re-add the books. Or if you just want to do it in bulk you can create a new empty library then use the add books -> add from folders -> one book per folder to recreate the library from your old library folder.

Today, 11:28 AM	#3
asw22 Junior Member Posts: 5 Karma: 10 Join Date: Aug 2024 Device: PocketBook InkPad 3	Thanks Kovid! I am trying the first method but the results window (which shows all the types of issue!) is blocked (not responding). I guess this is due to the very large size of the library and I just need to be patient.. I will try the second method later when Calibre will have given me control on operations...

Today, 12:04 PM	#5
asw22 Junior Member Posts: 5 Karma: 10 Join Date: Aug 2024 Device: PocketBook InkPad 3	I never added books in another way than via Calibre. I did however get many duplicates (for various reasons, but often because Calibre did not show me all the books!), and I removed some via Calibre and others directly in the library, but always by removing the containing folder of the book. For unfathomable reasons, I got myself many books without any format (about 10000 out of 50000) I am dumbfounded, to say the least!

Today, 01:14 PM	#8
asw22 Junior Member Posts: 5 Karma: 10 Join Date: Aug 2024 Device: PocketBook InkPad 3	All what I have described is happening on the same NVE SSD (and other internal disks in the same computer). I don't use cloud or NAS. I have done what Kovid suggested (create an empty library and filling it with the books of another one): this work fine except I los all the metadata painstakingly assembled/created from different sources (series, blurb...): this technique allows retrieval of the epub, but not the opf! How can I fix that? I would have 45000 sets of metadata to retrieve and adjust... PS I am playing with copies, the original library is intact!

Today, 04:51 PM	#11
Sirtel Grand Sorcerer Posts: 10,781 Karma: 226728226 Join Date: Jan 2014 Location: Estonia Device: Kobo Sage & Libra 2	It seems something has thoroughly messed up your library. What you're seeing is in no way normal. Are you sure you don't have any syncing or backing-up program running in the background? Of course, if you've been deleting books from your library folder without going through the Calibre GUI, that would also mess things up. But I think not as badly as you described.