Duplicate library mess

[travger]

Ok, saved to disk, but still not content. I have multiple pdf-s of the same book in the library, different only by metadata (like, the one I think best, has picture but as I have not yet converted and read it, just in case I keep others also, grouped nicely together in library view). Author and name are the same and so in the new folder I have only 1 pdf of any name - and must actually open to know which one. Not to mention that I'd like to keep others also.

For backup I've pretty much decided best procedure:
exit Calibre, navigate to library folder, dragdrop to another drive. Next time Windows asks to overwrite and that's ok.

[itimpi]

Originally Posted by travger

When I moved the library, I expected to be left with empty place where it was. Instead, on the first glance it was indistinguishable from the new library. So after some weeks and lot of clicking around ('what happens if I click this' type) I probably switched them around and added new books to both. Somewhen I forgot that I have 2 libraries and noticed it only recently. (Well, I expected 1 full and 1 empty)
What I'm trying to determine is, what did I do wrong and is there an easy way to merge them.

When you moved the library, then what is left behind is files that Calibre did not know about. If you look at the filing system level at a library that has just been moved to a new location and find a specific folder/file combination and then also look at the original library then you should find that folder/file is missing. Tidying up then involves working through these 'left behind' files to see if any of them need adding to the new library, and if not deleting them.

Note, however, this assumes that nothing went wrong during the move. If for instance you do not have delete rights on the original lib ray files then calibre will normally silently ignore the delete error. Tidying up is then much harder as it is no longer clear which files are the ones that need further examination.

[chaley]

Originally Posted by travger

Now I'm somewhat leery about just moving the library, I am fairly sure that some of the books I moved lacked some data or formats. Hence this post - if in the future I move x000 books, how can I be sure that not a single one has left anything behind.

This is a rather philosophical question.

What can be said is that calibre is not supposed to leave books/formats behind *that it knows about* when it moves a library. If it does leave such things behind, then there is a bug or permission problems on the file system.

Is there a guarantee of no bugs? No. Thus you cannot ever be certain. In fact, you can't be certain that tomorrow, calibre won't format your hard disk because gamma rays changed memory. That said, it is highly probable that calibre works well enough when moving libraries, and that it won't format your disk. Is that good enough?

I am not trying to be sarcastic. I just think that you aren't going to get the guarantees you are seemingly asking for.

Like, should 'check Library' window be empty when library is ok?

Depends on what you mean by 'ok'. That function tells you about 'things' that are in the library that calibre does not know about, and in the case of formats, things that should be in the library that aren't. Sometimes this is normal. For example, calibre2opds puts stuff into the library. MacOs fills it full of .DS_Store files/folders. Windows will drop little 'thumbs.db' files in some situations.

In other cases it is not normal. For example, finding extra formats would be strange. Finding authors or titles that calibre should know about (have not been deleted) would be strange. It is less strange (but still strange) to find authors or titles that should have been deleted, but weren't because of permission problems or having a format open in a viewer when the book was removed from calibre's database.

Only you can decide what is ok and what isn't.

BTW: nothing the checker reports should cause calibre to fail.

Can I correct things from there or is it just for info where the discrepancies are?

The function tells you things. It won't 'fix' anything, nor can you 'fix' things from there.

Then why all those clickable buttons in the bottom?

I don't understand this question. The buttons are there to do what they say. For example, if you change something (delete folders, add ignores, etc), you can push the run the check button to do it again. You can push the copy to clipboard button so you can preserve the report should you want to look around in calibre. Etc.

I confess that I have considered removing the 'Cancel' button because it does the same thing as 'Done', but that is a nit.

[chaley]

Originally Posted by travger

For backup I've pretty much decided best procedure:
exit Calibre, navigate to library folder, dragdrop to another drive. Next time Windows asks to overwrite and that's ok.

That is a rather inefficient way to do things, and also not quite correct. Inefficient because you are copying files that haven't changed. Incorrect because it won't get rid of folders and files that are no longer in the library.

Synchronizer programs can do the job well. The copy what is new/changed, and if instructed to, delete what isn't there any more. I have used synctoy, karens replicator, and xcopy. There are others.

You can also use windows backup. It does a tolerably good job for this sort of thing, and gives you some level of history.

[travger]

Originally Posted by itimpi

When you moved the library, then what is left behind is files that Calibre did not know about.

Thanks to this and some other threads I've come to understand it.

Originally Posted by itimpi

If you look at the filing system level at a library that has just been moved to a new location and find a specific folder/file combination and then also look at the original library then you should find that folder/file is missing.

If there's thousands of folders, how can I find the one or two that are wrong?

Originally Posted by itimpi

If for instance you do not have delete rights on the original lib ray files then calibre will normally silently ignore the delete error. Tidying up is then much harder as it is no longer clear which files are the ones that need further examination.

Very good notion! I checked the folder properties and it's read-only. I should have admin rights - (Win XP2) after all, nobody else uses my machine, I have no problems installing and deleting programs, but still can't uncheck Read-only. (It looks ok, but every time I open properties, Read-only is checked)

[travger]

Originally Posted by chaley

What can be said is that calibre is not supposed to leave books/formats behind *that it knows about* when it moves a library. If it does leave such things behind, then there is a bug or permission problems on the file system.

And I try to find out what happened and how to be *reasonably* safe, not asking any guarantees. But forewarned is forearmed.

Originally Posted by chaley

I don't understand this question. The buttons are there to do what they say. For example, if you change something (delete folders, add ignores, etc), you can push the run the check button to do it again. You can push the copy to clipboard button so you can preserve the report should you want to look around in calibre. Etc.

When I open Check Library, there's
Invalid titles
+Extra titles
Invalid authors
+Extra authors
Missing book formats
+Extra book formats
Unknown files in books

By clicking + I can see what is extra. Now I went to library folder and just deleted those folders. So, 'Run the check' is for the occasion when I have not closed the window before making changes? +signs went away.
If there no +, it means I have no invalid or extra titles?

[travger]

Originally Posted by chaley

Incorrect because it won't get rid of folders and files that are no longer in the library.

I hope to learn how to keep such things out of my library. When I moved it, apparently every folder and opf was copied and left behind.

Thanks on the tip on the synchronizers!

[chaley]

Originally Posted by travger

When I open Check Library, there's
Invalid titles
+Extra titles
Invalid authors
+Extra authors
Missing book formats
+Extra book formats
Unknown files in books

By clicking + I can see what is extra. Now I went to library folder and just deleted those folders. So, 'Run the check' is for the occasion when I have not closed the window before making changes?

Yes. Either after changes to the file system or changes to the contents of the ignore boxes.

+signs went away.
If there no +, it means I have no invalid or extra titles?

You have it right. If there are no + signs, then the report generator did not find anything that it wasn't told to ignore.

[chaley]

Originally Posted by travger

I hope to learn how to keep such things out of my library. When I moved it, apparently every folder and opf was copied and left behind.

This problem doesn't arise because of library/calibre issues. It arises because by using 'copy', the backup folder hierarchy is the union (a merge) of everything that has ever been in the library since the first copy.

Assume you make a copy today, and that the book The Doom of Horseflies is in your library. Tomorrow you delete the book, and the folders etc are removed from your library. You then do another copy backup. The book is not in your library and therefore won't (can't) be copied, but it is still in your backup because it was there yesterday. Your horseflies are still doomed.

There are some situations where a merge is exactly the behavior you want. For example, assume you keep 12 months of bank statements on your primary disk, but all bank statements on your backup media. Using copy would be appropriate, because removing older bank statements from your primary storage should not remove them from the backup media.

One must decide between backup and archive. A backup is a copy of something that can be used to recreate (easily) the original. An archive is a collection of data that is (usually) a superset of the original. Both schemes have their place.

(Apologies if I am explaining something you already know. I do tend to get pedantic. )

Thanks on the tip on the synchronizers!

You are welcome.

[Starson17]

Originally Posted by chaley

Assume you make a copy today, and that the book The Doom of Horseflies is in your library. Tomorrow you delete the book, and the folders etc are removed from your library. You then do another copy backup. The book is not in your library and therefore won't (can't) be copied, but it is still in your backup because it was there yesterday. Your horseflies are still doomed.

An even more common problem in Calibre is fixing title or author errors. Suppose you realize that you misread the title of the book when you first entered it and it was stored as "The Dome of Horseflies." You fix it to the "Doom" but now your horseflies are both doomed and domed in the backup. The same problem occurs when you fix an author. I have 20 books by one author, but the name may be stored as A.C. Clarke, Arthur Clarke, AC Clarke, A C Clarke, etc. When I spot that error, I like to get all the author names stored consistently, but using "Copy" for the backup leaves all the old versions, as well as the new ones.

I use a one-way mirror synchronizing program to keep my library backup.

[travger]

Better pedantic than unclear. I would expect my backup copy reflect exactly current state of my library - some book has only cover, another only metadata, deleted books vanished without a trace - so after restore I can go on like I had done no restore at all. I do not want to have delete books I already deleted. (Original files are anyways separate from Calibre)

[travger]

I got rid of all the + signs, library is checked.

I copied books (formats? - got no notice about 'you already have opf') from new location back to the old location. As much as I understand, there's no way to make sure that both libraries are identical, exept to compare them manually?