[GUI Plugin] Find Duplicates

[komali_2]

Originally Posted by random733

Is there any working version with the auto merging feature? komali2s version (1.8.5) here unfortunately does not work with the latest Calibre version (linux and windows) - keeps requiring a reboot without the plugin being fully installed.

I understand the concerns about the dangers to the integrity of the library. But I think you can manage the risk with a backup. And in the end, everyone is responsible for actually using this function (which is also explicitly declared as dangerous).

Hey, sorry about that, I got caught up in some big work projects.

It's ok that kiwidude doesn't want to include a fork's functionality, that's very normal in the open source community for people to have strong opinions - such things are required for really cool projects like theirs to get created for free. I don't actually disagree with their logic either, combining duplicates carries high risk of screwing up your ebooks and metadata and losing copies! You have to be careful.

But, there's still usecases. I had one, so I made the tweak to the plugin. Others have usecases too. I'm working right now to get it working with up to date versions of calibre. There was someone else that kindly emailed me to mention another bug that I'm looking at.

As for @kiwidude, don't worry, I won't be making a PR, though i appreciate you linking https://github.com/kiwidude68/calibr...icates-v1.10.7 your github repo, there's lots of really good information there. The primary motivation for me to create my tweak was to try doing some open source work outside of web dev, thanks for creating resources to help people do that! As your code is GPL-3, I'm going to change my repo to be a fork of yours with the same license. I really like GPL-3 as well.

[komali_2]

Sorry to double reply, but it looks like there's a time limit on editing a post.

The fork of kiwidude's excellent repo for finding duplicates that I made a while back has been updated to (hopefully) work on newer versions of Calibre, and also includes the massive improvements kiwidude made to their code in the meantime. https://github.com/komali2/calibre-find-duplicates

Remember that this is only possible because of all of their hard work, credit goes to them!

[BetterRed]

Moderator Notice

@komali_2: See this regarding edits and PMs.

BR

[feuille]

Hello kiwidude,

when Calibre was running in debug mode today, I noticed that an key error is reported in the book_algorithms.py module:

File "calibre_plugins.find_duplicates.book_algorith ms", line 295, in _find_candidate_by_file_size
KeyError: 'mtime'

Statement 295 in version (1, 10, 7) is: mtime = stat_metadata['mtime'].

However, the plugin continued to run and returned a result.

Regards,

[kiwidude]

Originally Posted by feuille

Hello kiwidude,

when Calibre was running in debug mode today, I noticed that an key error is reported in the book_algorithms.py module:

File "calibre_plugins.find_duplicates.book_algorith ms", line 295, in _find_candidate_by_file_size
KeyError: 'mtime'

Statement 295 in version (1, 10, 7) is: mtime = stat_metadata['mtime'].

However, the plugin continued to run and returned a result.

Regards,

Hi fueille,

The only way I can see this happening is if you somehow have invalid metadata in your library for a book format. Perhaps you deleted a book format from disk, but calibre still thinks it exists.

I will tweak the code so the exception won't appear in future, but with the way it is written it would just skip that book format anyways which is what it should do, it is just extra noise in the log.

[kiwidude]

Release Notes
https://github.com/kiwidude68/calibr...icates-v1.10.8

[ownedbycats]

Question: Is it possible to run a binary compare on just a few selected books, instead of the entire library?

[kiwidude]

Originally Posted by ownedbycats

Question: Is it possible to run a binary compare on just a few selected books, instead of the entire library?

Yeah just make a virtual library with those books on it, Find Duplicates respects virtual libs.

[Wiggo]

Originally Posted by capink

Unfortunatly, this is not possible. Custom columns in calibre do not support ampersands as a separator. I opened a bug report a while back about this issue, but it was not addressed because of technical difficulties.



The solution for your problem is to do away with ampersands as separator and fall back to using commas. You can do this by modifying the template function as below

def evaluate(self, formatter, kwargs, mi, locals, val, col_name, user_cat_prefix):
    new_val = ''
    if hasattr(mi, '_proxy_metadata'):
        all_cats = mi._proxy_metadata.user_categories
        cats = {k:v for k,v in all_cats.items() if k.startswith(user_cat_prefix)}
        SEP = mi.metadata_for_field(col_name)['is_multiple'].get('list_to_ui', '')
        new_val = set()
        if SEP:
            val_ = val.split(SEP)
        else:
            val_ = [val]  
        for user_cat, v in cats.items():
            repl = user_cat.lstrip(user_cat_prefix)
            for user_cat_item, src_cat in v:
                if src_cat == col_name:
                    for item in val_[:]:
                        if item == user_cat_item:
                            new_val.add(repl)
                            val_.remove(item)
                        else:
                            new_val.add(item)
        if new_val:
            return ', '.join(list(new_val))
    return val

Note, however, that this will break author names with commas in them.

Unfortunately the problem is back. Two names separated by "&" are treated as one again

Edit: Nevermind. I swapped to this method, which works a bit better in my eyes.

https://www.mobileread.com/forums/sh...77#post4333077

[kjdavies]

Originally Posted by kiwidude

By comparison, with this plugin I can test 40000 books in under a second and once my exemptions are in place any future comparisons will take negligible time to perform and maintain.

Incidentally, I've taken to updating my files with a couple of new 'identifiers'.

One is SHA, the other is called 'CRC' but lies a little (it's actually CRC32-<type>-<size>). Your lovely plugin here makes it very easy to reliably dedupe based on this information.

I still use a binary dupe check on occasion to look for files with different extensions (I found one set that had 'XML' and 'SRD' extensions -- one was the actual format, the other indicated the data set), but just having this is very handy.

I can then, in a dedupe cleanup library, present the 'CRC identifier' in the tag browser. They're grouped by first character, giving me 16 groups of CRC values to quickly skim looking for 'more than 1' entries and cull. It's a little more convenient for me than the options built into the plugin.

[ownedbycats]

How does "Similar" in Find Metadata Variations work? I can't remember the last time I even got a match with it, so I'm not even entirely sure it does work.

[kiwidude]

Originally Posted by ownedbycats

How does "Similar" in Find Metadata Variations work? I can't remember the last time I even got a match with it, so I'm not even entirely sure it does work.

As per the description on the dialog. For a similar title it removes punctuation characters like \(){}<>'";,:#-._ and certain common words from the title like a/the/an

For similar authors it strips initials, punctuation and ignores order of the names so John Smith and Smith John will match.

[maripopa]

Hello guys,
I am using calibre as an e-book/e-documents for many years.
Now I am facing a new challenge:
I merged two libraries (medium size, both). The new resulted library contains more duplicates than I expected initially.
The problem:
I run binary compare to find real doc duplicates (without option to delete duplicates formats during operations). The result is a quite big list/groups.

Can I save the result for processing the metadata/formats merge AFTER, without need to rerun the duplicate finder again (my this case is a long operation (the library is medium size 500GB and ~200k entries))

Any help/suggestion welcomed
Thanks for your time!

[BetterRed]

Originally Posted by maripopa

Can I save the result for processing the metadata/formats merge AFTER, without need to rerun the duplicate finder again (my this case is a long operation (the library is medium size 500GB and ~200k entries))

How about putting the duplicates into a Reading List ==>> plugin

BR

[luciaisacat]

Originally Posted by maripopa

Can I save the result for processing the metadata/formats merge AFTER, without need to rerun the duplicate finder again (my this case is a long operation (the library is medium size 500GB and ~200k entries)

In these cases, I just add a temporary tag to the resulting titles so that I could call it back when I wish to work on those entries. Too simple?
L