Scan for duplicates

[lbutlr]

Probably as a result of an rsync error at some point, I have a lot of duplicates in my library, and cannot find any way to automatically find them all and merge them. I've seen mention of a "Merge Duplicates" plugin that doesn't seem to exist anymore, and I found a thread that is many years old.

https://www.mobileread.com/forums/sh...d.php?t=255394

But it refers to the plugin. I've scoured the menus and the preferences, but I am not seeing anything helpful. I estimate that I have at least 600 books which appear twice in my library, so I do not wan to do this by hand.

Short of creating an entirely new library and importing all 9000 books, is there anything I can do with the current version of Calibre to fix this?

[ilovejedd]

Merging book records is a built-in function of Calibre (keyboard shortcut M).

First you select the book record you want to keep, then select all other records you want to merge into that and press M.

[lbutlr]

Sorry, "Find Duplicates" was not listed on the plugins page and I didn't remember that the "Find plugins" button seached more plugin sources. I've found the plugin and am taking a look at it now.

[lbutlr]

Originally Posted by ilovejedd

Merging book records is a built-in function of Calibre now (keyboard shortcut M).

First you select the book record you want to keep, then select all other records you want to merge into that and press M.

That would require me to select two books and hit "M" at least 600 times.

[ilovejedd]

Originally Posted by lbutlr

That would require me to select two books and hit "M" at least 600 times.

There's no completely automated way to do it.

The "Find Duplicates" plugin only identifies the duplicates. It doesn't delete them (unless they're binary identical).

[theducks]

There is no SAFE, completely automated way to do this.
Find duplicates doe get false positives. That is why it offers more than 1 'dup' detector.
Nothing beats the ole' Eyeball Mk 1.
And what if they are 'almost' dups, that you want to keep BOTH? (orig edition, 2nd edition? English and French?)
Maybe if you were not in such a rush to begin, you would not be spending all this effort cleaning up?
One of the first things I did with Calibre, was document my 900+ paperback collection. It took me over a year of, a dozen at at a whack, metadata cleanup.

GIGO

[Tarana]

Originally Posted by lbutlr

That would require me to select two books and hit "M" at least 600 times.

Find duplicates just finds the duplicates. Then you have to perform an action on 1/2 of them (either merge or delete). Good project while you are on hold. Happened to me once resulting in over 1000 duplicates. Currently combining a couple of libraries (created before Virtual was available or before I understood how it worked). Time consuming but more productive than doodling while on hold.

Also, it bears to consider checking each book before deletion. On one of these libraries, I discovered a bunch of books that still had DRM - when BooksOnTheKnob had lots of freebees. Some of them I am getting resurrected on Amazon so that I can reload into Calibre, but some I'm just out of luck. Plus some I am no longer interested in. No guilt- these were free.

[yonkyunior]

in add books setting, enable auto merge set to overwrite.

[lbutlr]

It's a shame that I can't use tow features that make a computer super useful (telling when things are identical and telling when they are nearly identical) to automate a task instead of doing it by manually clicking, but I have cobbled together a different solution and created a new calibre library that is free of duplicates in several orders of magnitude less time than it would have taken me to manually sort through the dupes in calibre.

Still, it seems a shame that "this might not be safe" is treated as "No, no matter what, you'll have to do this the hard way". Sure, put up warnings, but if I want you to automatically-merge anything with identical titles and names without starting over, let me.

[mbovenka]

If titles and authors are indeed *identical*, dup checking on import will catch them. Create a new library, set, like yonkyjunior says, 'automerge' to 'overwrite' and copy all books in your existing library to the new one.

Duplicates will merge automatically.

[lbutlr]

Originally Posted by mbovenka

If titles and authors are indeed *identical*, dup checking on import will catch them. Create a new library, set, like yonkyjunior says, 'automerge' to 'overwrite' and copy all books in your existing library to the new one.

Duplicates will merge automatically.

This is what I ended up doing, though it meant rebuilding the entire library and redownloading meta data. and "identical" since cources cannot agree on naming conventions. Importing the library with the metadata failed because that got me back to exactly what I had before, with duplicate books listed as separate entries.

C S Lewis, C S Lewis, Clive S Lewis, Clive Staples Lews, Clive S. Lewis. It's even worse for JRR Tolkien since JRR, J R R, J. R. R are all frequent variants, along with various random seeming expansions of his names.

To say nothing of the disaster that is marking an editor as the author which ads many more variations to try to sort through. I must have seen 50 variation on the tags for books edited by George RR Martin.

Heck, even with Jane Austen I had one book with the author tagged simply as "Austen" one a J. Austen, and one tagged as J Austin. ¯\_(ツ)_/¯

(Since the metadata doesn't modify the original book, I essentially had to start all over from the original sources)

[DNSB]

Originally Posted by lbutlr

This is what I ended up doing, though it meant rebuilding the entire library and redownloading meta data. and "identical" since cources cannot agree on naming conventions. Importing the library with the metadata failed because that got me back to exactly what I had before, with duplicate books listed as separate entries.

C S Lewis, C S Lewis, Clive S Lewis, Clive Staples Lews, Clive S. Lewis. It's even worse for JRR Tolkien since JRR, J R R, J. R. R are all frequent variants, along with various random seeming expansions of his names.

To say nothing of the disaster that is marking an editor as the author which ads many more variations to try to sort through. I must have seen 50 variation on the tags for books edited by George RR Martin.

Heck, even with Jane Austen I had one book with the author tagged simply as "Austen" one a J. Austen, and one tagged as J Austin. ¯\_(ツ)_/¯

(Since the metadata doesn't modify the original book, I essentially had to start all over from the original sources)

That is why quite a few people use a Intake library. An ebook is imported to the Intake library, the metadata is updated and massaged to match your standards, the modified metadata is added to the ebook and only then is it moved to the main calibre library. For the examples you gave, I would use bulk metadata edit to set the author name/sort for consistency.

A pain if you already have a large collection but once you have gone through the pain, it makes for a much cleaner and easier to search library.

[ilovejedd]

Originally Posted by DNSB

For the examples you gave, I would use bulk metadata edit to set the author name/sort for consistency.

I find standardizing authors, tags, etc. much easier using "Manage {field}" in Tag Browser rather than Bulk Metadata Edit.

[DNSB]

Originally Posted by ilovejedd

I find standardizing authors, tags, etc. much easier using "Manage {field}" in Tag Browser rather than Bulk Metadata Edit.

In the main library, it might be easier but given the small size of my Intake library, BME works for those rare occasions where I want to modify more than one book's author.

[BetterRed]

In the Intake library I don't normally bother with Manage or BME. I edit cells in the book list, and if its worthwhile (e.g. several books in a series) use Copy & Paste to copy columns from book A to books B, C and D. For me, jumping in and out of dialogue boxes is time wasting - I also find it very bureaucratic.

BR