Unable to restore databasecalibre

[asw22]

Hello everyone!

An thanks to Kovid for this wonderful too that is Calibre!

I have been a Calibre user for about ten years (now 7.15/W11), and I manage about 45000 books, without too many issues...However the issues I have had recently are a bit strange:

1) on my main library, when I make a research via author or series, some books are missing for no reason...and yet they are in the library folder in their proper place, and so far (say one year ago) there was no issue...
So I tried to restore the database...same issue!

2) I made an experiment and duplicated the Calibre folder under another name via the export/import process using an intermediary folder...I obtain an exact duplicate of the original database, with the same books still not considered in the metadata.db

3) Having a full library to play with, I try to restore the database: this time the original database (75M) is "destroyed" (albeit saved) and replaced by a 5 book database or so (404K)!!! I try again, same thing!

4) I try a last test, and remove the metadata.db, hoping the restauration will force Calibre to recreate it via the books data in the Calibre folder (like it is supposed to do), thus allowing me to get all the books: no dice I still get the same small database!!!

What could happen?

Thanks for your lights!

asw22

[kovidgoyal]

Use the check library tool to find all such problems and then you can re-add the books. Or if you just want to do it in bulk you can create a new empty library then use the add books -> add from folders -> one book per folder to recreate the library from your old library folder.

[asw22]

Thanks Kovid!

I am trying the first method but the results window (which shows all the types of issue!) is blocked (not responding). I guess this is due to the very large size of the library and I just need to be patient..

I will try the second method later when Calibre will have given me control on operations...

[Sirtel]

You should also try to find out what has caused your database issues, to prevent them from happening again. Is your library on a cloud or a network drive? Have you added/deleted books directly from the library folder, without doing it via the calibre GUI? Those are the most common reasons for data corruption.

[asw22]

I never added books in another way than via Calibre.

I did however get many duplicates (for various reasons, but often because Calibre did not show me all the books!), and I removed some via Calibre and others directly in the library, but always by removing the containing folder of the book.

For unfathomable reasons, I got myself many books without any format (about 10000 out of 50000) I am dumbfounded, to say the least!

[Sirtel]

Originally Posted by asw22

...and others directly in the library, but always by removing the containing folder of the book.

Never do that. The book entry remains in the database that way.

Any chance of cloud syncing? OneDrive, Google Drive, an automatic backup solution?

[theducks]

Originally Posted by asw22

I never added books in another way than via Calibre.

I did however get many duplicates (for various reasons, but often because Calibre did not show me all the books!), and I removed some via Calibre and others directly in the library, but always by removing the containing folder of the book.

For unfathomable reasons, I got myself many books without any format (about 10000 out of 50000) I am dumbfounded, to say the least!

2-way SYNC software is the most likely cause.
You fix a title or Author and the sync puts the old one back. OR you use Google Drive (which damages the file name ) Do read the FAQ about NAS shares.

[asw22]

All what I have described is happening on the same NVE SSD (and other internal disks in the same computer). I don't use cloud or NAS.

I have done what Kovid suggested (create an empty library and filling it with the books of another one): this work fine except I los all the metadata painstakingly assembled/created from different sources (series, blurb...): this technique allows retrieval of the epub, but not the opf!
How can I fix that? I would have 45000 sets of metadata to retrieve and adjust...

PS I am playing with copies, the original library is intact!

[JSWolf]

Originally Posted by asw22

All what I have described is happening on the same NVE SSD (and other internal disks in the same computer). I don't use cloud or NAS.

I have done what Kovid suggested (create an empty library and filling it with the books of another one): this work fine except I los all the metadata painstakingly assembled/created from different sources (series, blurb...): this technique allows retrieval of the epub, but not the opf!
How can I fix that? I would have 45000 sets of metadata to retrieve and adjust...

PS I am playing with copies, the original library is intact!

Select the original library. In the toolbar lick on the library icon. Then select Restore Database. That will build a new database from your eBooks in the library. I'm not sure if the option to check the library comes up after that. If it does, do it. If not, select Library maintenance and check the library.

[asw22]

I did that, and the original library database was "properly" restored, with the same issues as before:

1) I have a list of a dozen books without format...for instance I have a series of 14, for which there are 14 folders in the library including two without format: only two books shown in Calibre, and the two without format also shown?? It totally baffles me!

2) another series of 12, all ok except number 2, which is properly present in the library with its 11 other issues, except this one is not shown by Calibre!

I have no idea what happens...

3) I have many duplicate folders, one complete, one with just the opf. The verification of library seems to identify them, but I seem unable to use the verification process properly: even after 1/2h waiting the process is still non responsive...

I wonder if a interesting way to recreate a library would be to use an improvement to the technique mentioned by Kovid of extracting only books from the source library (which is the case now with the loss of all cover & metadata), but the individual book folders, including cover and metadata, with checking there is a real book inside? Is there already a way to do that?

Thanks to all of you helping me!

asw22

[Sirtel]

It seems something has thoroughly messed up your library. What you're seeing is in no way normal. Are you sure you don't have any syncing or backing-up program running in the background?

Of course, if you've been deleting books from your library folder without going through the Calibre GUI, that would also mess things up. But I think not as badly as you described.

[JSWolf]

Originally Posted by Sirtel

It seems something has thoroughly messed up your library. What you're seeing is in no way normal. Are you sure you don't have any syncing or backing-up program running in the background?

Of course, if you've been deleting books from your library folder without going through the Calibre GUI, that would also mess things up. But I think not as badly as you described.

Something has to be doing this. It doesn't just happen.

s @asw22 going into the library and changing things? I agree that it could be some sort of sync program maybe syncing to/from Google Drive or some sort of corrupt library. I don't know what gets corrupted from Google Drive. I just know it does get corrupted.

I can't see it being an AV program causing the problem. But if it is, that AV program would have to go.

The thing to do (IMHO) is to copy the existing library and clean up the library and get it as good as it can be and then do a proper backup. But before doing that, put the library someplace else so if it is a sync program, it won't sync the new library.

[Sirtel]

Originally Posted by JSWolf

Something has to be doing this. It doesn't just happen.

s @asw22 going into the library and changing things? I agree that it could be some sort of sync program maybe syncing to/from Google Drive or some sort of corrupt library. I don't know what gets corrupted from Google Drive. I just know it does get corrupted.

It doesn't need to be Google Drive. Any sync program that runs in the background all the time could do this.

[BetterRed]

@asw22 - what OS are you using (Windows, MacOS or Linux)?

What is the path to your library folder?

By default Windows auto-syncs the Desktop, Documents and Pictures to Onedrive. I think that started within the past ten years. It's best to keep calibre library folders well clear of those locations - e.g my libraries have paths such as:

D:\Calibre\Libraries\Books 
C:\Users\xxxxxxxx\CalibreLibrary.

BR

[asw22]

Originally Posted by BetterRed

@asw22 - what OS are you using (Windows, MacOS or Linux)?

What is the path to your library folder?

By default Windows auto-syncs the Desktop, Documents and Pictures to Onedrive. I think that started within the past ten years. It's best to keep calibre library folders well clear of those locations - e.g my libraries have paths such as:

D:\Calibre\Libraries\Books 
C:\Users\xxxxxxxx\CalibreLibrary.

BR

Hello everyone!

My libraries are on various disks and the basic ones are on C:/ in a dedicated directory.

I had fun (NOT) removing from yhe library folders with only the opf file in it....I have removed about 6 ot 7000 (!!!!), plus some real duplicates at folder level (but with different nr).
After that I removed the metadata.db, and restored the database.
The result seems fine and works well (46000 books) but for some reason, some books of some series (not all of them mind you!) are still not displayed by Calibre, even as their are present in the library in a proper way. Out of 7, I have 3 shown on this example!
I removed the db again and restored it again: same issue with the same books...

I have tried to reintegrate one of the no-shows (and removed the original folder, which was not shown anyway) , and after restoration of db, again not shown, with the three others still shown...

Black Magic!