[GUI Plugin] Find Duplicates

[Narcio]

A binary compare for duplicates during merging is exactly what I thought would be helpful to avoid all of the trouble I'm having now. So yes, I see your point on the merge plugin side of things.

In the second paragraph you said that you think its unnecessary for a merging plugin to allow a user to look and choose between metadata from either of the potential duplicate books. I think this actually would be a useful option, and a merge plugin like what you described with this feature would solve 99% of my problems before they happened.

I agree with you that the root of the problem is in the merging.

Your plugin is extraordinarily useful nonetheless.

[kiwidude]

Originally Posted by Narcio

In the second paragraph you said that you think its unnecessary for a merging plugin to allow a user to look and choose between metadata from either of the potential duplicate books. I think this actually would be a useful option, and a merge plugin like what you described with this feature would solve 99% of my problems before they happened.

My concern with trying to offer flexible options for merging metadata would be coming up with a UI design which doesn't make it all too horribly complicated to use.

Not that I would completely ignore metadata with an initial plugin version of course (at risk of sounding like I am writing this here but I'm not - well not yet at least!). Perhaps a few (state persisted) checkboxes could give a little more control/flexibility over what you have in Calibre today.

For instance I know Starson17 loves and defends this behaviour, but I disagree with the concept of merging comments where they differ (rather than just where there are none). There are also scenarios where people do not want to merge tags. I think these two stand in isolation from the rest of merging metadata, as everything else is considered as a "replace if empty".

Starson17 was kind enough to put in a "merge formats only" option into Calibre to replace my own local patching of the code. I think this smart merge plugin could just have a few checkboxes in it giving options like "Merge formats only" and if unchecked some granular choices of merge behaviour for the comments and tags fields. I think perhaps something like that would satisfy the 98% scenario.

You could potentially also consider covers - should the user be able to choose which of multiple covers should survive, or should it just that of the "master" (first selected record chosen before starting the merge operation).

It's a slippery slope and a challenge I think to find the right balance between keeping the merge sufficiently simple and non-intimidating versus flexibility. The existing merge operations in calibre choose the "simple" route, with some flexibility provided by mutually exlusive merge menu options. I would like a smart merge plugin to act as a superset of those in a single dialog. A power user who knows that they want to do a merge copy or whatever can still use the existing calibre menu options, but anyone else who wants a bit more control and assistance with understanding which format of multiple to keep etc could use the smart merge plugin.

As with a number of my posts, it is just me thinkout out loud at this point... I'm glad you are finding the duplicates plugin useful anyways.

[Partzz]

Thank-you. Plug-in works FAST. Did a similar search using title and author, took about 10 seconds to go through 24,000+ titles with 360 dupes found! Good Job!

[speakingtohe]

Perfection. Thank you
Helen

[geordiejohn]

after reading through all the topics i have finally got it to work and it is now showing on my toolbar,thank you very much.

[geordiejohn]

i spoke too soon,when i connect my Kindle the find duplicates goes away,i have tried to setup up so when Kindle is connected to show duplicates.
any ideas what i am doing wrong please as i would like to find duplicates on my Kindle ?
thank you.

[kiwidude]

Go to Preferences->Toolbar, and change the dropdown on the top right to "The main toolbar when a device is connected". Or similarly for the context menu if that is where you put it. There are different toolbars/menus that you customise based on whether a device is or is not connected.

[geordiejohn]

i went to pref-toolbar and main toolbar when device connected but i cannot apply as it is highlighted,i have to cancel ?

[kiwidude]

Once you have switched the dropdown to "Main toolbar when a device is connected", you still need to actually make changes to that toolbar before the Apply button will not be greyed out. So you will want to select the "Find Duplicates" plugin from the left side of Available Actions, click the right arrow to put it onto that particular toolbar you chose in the dropdown. Once you have done that you will be able to click Apply.

[geordiejohn]

by jove i think i have done it.
thank you.

[drMerry]

Hi Kiwidude,

An interesting error:
I did a bit compare and found these duplicates:


The interesting part is, I had 500 duplicates. I selected every first one and deleted them (it's bit, so true-duplicate).
When I found these duplicates, I got curious because of the many dups. I did see the different size of books and did an external duplicate check.
It occurred that only the epubs are duplicates.

Is there a way to tell the user there is no complete match on all formats, or to show which formats are dups?

Some cases with (by me) desired output (assume same type is duplicate):
Book A - txt
Book A - txt, epub
Book A - epub

return on dupcheck:
Book A - txt
Book A - txt, epub - 50% match
Book A - epub

return on dupcheck of txt files:
Book A - txt
Book A - txt, epub - 50% match

So, when 100% dup, no extra info needed, otherwise there is more info
An other option could be to create several filters:
marked:complete_duplicate_group_x
markedartial_duplicate_group_x
marked:all_duplicate_group_x

[kiwidude]

@drMerry - I believe this is the same issue that I have recently discussed in several posts starting at the end of the previous page. This is a merge issue, not a duplicate finding issue. Duplicate find has done its job and told you which books are duplicates. Whether they contain duplicate formats or complementing formats and how you get those down to a single book record is now all about the merging, which I posted my thoughts on.

[drMerry]

You are right about that.
I was only thinking about a possibility for extra warnings at this point. But it will be a hard nut to crack I suppose. And at the other hand, most of the functions (except this one) show POSSIBLE dups, so it is up to the user to check...

[htweedie]

I seem to be having an error when I try and run the plugin in Calibre 0.8.2 on OSX.

It seems to have installed okay (find the installer plugin) and on the title bar.

When I click the title bar I get the error RuntimeError: maximum recursion depth exceeded while calling Python object with the following traceback/details:

Appreciate any help :-)

calibre, version 0.8.2
ERROR: Unhandled exception: <b>RuntimeError</b>:maximum recursion depth exceeded while calling a Python object

Spoiler:

Traceback (most recent call last):
File "site-packages/calibre/gui2/library/views.py", line 765, in search_proxy
File "site-packages/calibre/gui2/library/models.py", line 298, in search
File "site-packages/calibre/library/caches.py", line 773, in search
File "site-packages/calibre/library/caches.py", line 791, in search_getting_ids
File "site-packages/calibre/utils/search_query_parser.py", line 193, in parse
File "site-packages/calibre/utils/search_query_parser.py", line 201, in _parse
File "site-packages/calibre/utils/pyparsing.py", line 1089, in parseString
File "site-packages/calibre/utils/pyparsing.py", line 962, in _parseNoCache
File "site-packages/calibre/utils/pyparsing.py", line 2681, in parseImpl
File "site-packages/calibre/utils/pyparsing.py", line 962, in _parseNoCache
File "site-packages/calibre/utils/pyparsing.py", line 2536, in parseImpl
File "site-packages/calibre/utils/pyparsing.py", line 962, in _parseNoCache
...
File "site-packages/calibre/utils/pyparsing.py", line 293, in __init__
RuntimeError: maximum recursion depth exceeded while calling a Python object

[drMerry]

Is it possible to change settings on preferences->(advanced) Plugins->User Interface Action Plugins->Duplicate Check

===

EDIT (I did mean Duplicate off-course)