Duplicate Detection

[kiwidude]

Originally Posted by Starson17

I'm not sure I understand that last part - but perhaps it's clear from the above. It doesn't offer individual book by book control, but does (will) create duplicate records.

Your explanation above was sufficient. What confused me was your original explanation, which to paraphrase sounded to me like the three options were: Kovid's legacy code, automerge behaviour as it is now, and overwrite existing with new dupe formats. There was no mention of the new books for dup formats option from that which is why I wondered if the idea had been dropped.

However your responding post was much clearer and sounds to me the same as what we have been proposing in earlier posts, just with a combobox of automerge suboptions instead of radio buttons. Great stuff.

[kovidgoyal]

@Starson17: I find I like to work in bursts, surfing MR is a good way to let my brain degauss between coding sessions

Yes, that should work. I dont recall if it is (translated string', 'option') or ('option', 'translated string') but a bit of experimentation will tell you

[Starson17]

Originally Posted by kovidgoyal

Yes, that should work. I dont recall if it is (translated string', 'option') or ('option', 'translated string') but a bit of experimentation will tell you

I think I've got it right - but as usual, I get to leave it to you to make sure

[Starson17]

Originally Posted by kiwidude

Your explanation above was sufficient. What confused me was your original explanation,

Sorry I wasn't clear

However your responding post was much clearer and sounds to me the same as what we have been proposing in earlier posts, just with a combobox of automerge suboptions instead of radio buttons. Great stuff.

The code is done, except for cleaning up the label text and tooltips and one detail. What do I do for the "New Record" option when I have multiple matching records?

The functionality for multiple matching records hasn't been mentioned here before, but in the existing automerge code (to be the "Ignore" option in the new code) we put a new incoming format into all matching records, if possible, and ignore it, if not.

For "Overwrite" I'm going to overwrite or add the format to all - again that's easy and already done.

The question is: what to do with the "New Record" option for multiple matching records?
Suppose I have 5 book records already in the database, all with the same author/title, but some with the same format as the incoming format and some without. Is this a "create a new record" situation or not?

1) Shall I add the new incoming format to the records that don't have it and create exactly one new record?

2) Or do I not create the new record because the incoming format was already added to at least one existing book record?

3) Or do I create one new book record every time I find an existing book record I can't add the new format to (but add it to all records where possible) - thereby creating multiple identical new book records.

4) Or do I create one new record and not add it to any existing records?

5) Or do I create one new book record every time I find an existing book record I can't add the new format to (and not add it to any existing records).

I lean towards option 1, but option 3 is how the new code is working now. Option 3 is stateless - as in I don't need to keep track of what I did (or will do) for other matching book records. I just check each matching book entry, compare the new format to the formats in that record and create a new record if I can't stuff the incoming format into the matching book record.

Note that for option 3 you automatically get one independent record for every case where you already had that format in another record. You can then work your way through that list and manually "Merge-Delete Others" each record into the existing record that caused it to be created (where you wanted overwrite) or just delete it (where you wanted ignore) or just manually delete all but one (if you like option 1)

Can I convince anyone option 3 is the best .... it's much easier - because it's already done.

Comments?

[kovidgoyal]

I'd say 1 sounds right

[Starson17]

Originally Posted by kovidgoyal

I'd say 1 sounds right

Done. I'll try to clean up the text, tooltips, etc.then upload it this weekend for the normal refactoring

[Starson17]

Originally Posted by Starson17

Done.

New code is uploaded.

[kiwidude]

Originally Posted by Starson17

New code is uploaded.

Thanks Starson17, look forward to playing with it when Kovid has merged it in.

Guess that then leaves the onus next on me . I will need to change the Quick Preferences plugin to support the new automerge options.

Then there is the whole issue of the "Duplicate Find" plugin. Charles had some really good food for thought on this that he sent me via e-mail which I need to digest on. I would rather he posted them here himself or ask him for permission to mention them as they are certainly worth discussion as to which approach to take for the plugin.

[Starson17]

Originally Posted by kiwidude

Thanks Starson17, look forward to playing with it when Kovid has merged it in.

He's fixed it up and merged it into the trunk. Feel free to test it, if you can run from code.

[kiwidude]

Originally Posted by Starson17

He's fixed it up and merged it into the trunk. Feel free to test it, if you can run from code.

Awesome - I've got it now.

I wanted to respond to a comment I made/Starson17's response in a different thread here for further discussion:

Originally Posted by Starson17

That's fine by me, but remember that the reason I didn't fuzzy match more aggressively is that automerge is AUTOmatic. An error in automerge meant a lost book format. If we are doing duplicate finding with manual review on a dialog screen with merging manually controlled to find the best format or not merge, we can be much more aggressive.

My emphasis was intended on "1.0" and "may" as it was inline with some of the ideas we have discussed on this thread. Agree totally that we will want at least longer term to have fuzzy matching on author and "fuzzier" matching on title. As you say if the user has an interactive opportunity to view formats and reject false positives in their own time then "fuzzy" becomes a good thing.

In my mind these are the reasons why people may be in a duplicate situation (there could well be others I haven't thought of):

1. users who have never discovered the automerge option - so this will be their first chance to try a duplicate comparison on their library but in a non-destructive fashion.
2. users who knew about it but kept it turned off because they didn't want duplicate formats automatically being discarded before they could view them
3. any situation in which the user edits the title or author - the result may make the book a duplicate of another record. They may have had automerge turned on when they added it, but their filename did not match the regular expression or was not "exact" enough for the logic to work at that time
4. users who get the next version of Calibre using the new "Create new record for duplicate format" option with automerge
5. users who have had automerge on, but had variations in the title that exceed the current "fuzzy" matching logic it has. Similar to #3 above, except the user has not yet identified that the title should be edited.
6. users who have duplicated authors due to very slight variations in their names, such as spaces, variations of first names/initials/punctuation etc. As the current automerge is (very rightly) conservative of doing exact match only on author, the slightest difference results in a duplicate.

So an initial duplicate finder release that reuses the exact same comparison logic automerge has today (exact match author, fuzzy title) will help out all but the last two situations. Which imho is the 90% scenario for users, but I have absolutely no facts to support this so feel free to disagree. It is certainly the easiest use case.

Also by keeping the same conservative matching logic, a user can have confidence that the duplicate results list that appears are "genuine" duplicate scenarios. Well unless their title or author is completely wrong of course, but that is something only visual inspection of the book format can identify.

The results of a duplicate search using this logic may contain groups that require less manual visual inspection to merge together than others (i.e. groups which have multiple books but not duplicate formats, and groups that do have duplicate formats). However there are still problems which prevent the plugin from being "automatic" in merging those "safer" ones. As Starson17 at least is aware the problem is the book metadata. If you have only just added a duplicate to your library, then the "oldest" by book date duplicate is most likely (but not always) the one which contains the metadata you want to keep (series information, comments, ISBN, cover, conversion settings etc). That behaviour would effectively match what happens with automerge turned on today when adding books and you add new formats of a book.

However what if (as frequently happens) both book records have been in your library for a while, so both have metadata assigned but they differ in content? Maybe they have different series names (or one has one, the other doesn't), etc, etc. That is why there are so many "merge" submenus - as a user you have been given the power to merge to cover lots of scenarios of wanting only certain data kept with a particular merge direction.

Which (slowly, sorry) brings me back to my initial suggestion in the thread of wanting to use the power of the library view for the plugin (rather than a popup dialog displaying duplicate search results). Doing so means the user has the full merge menus that exist and they know today. They can pop open the edit metadata dialog or make changes directly in the grid before they merge. They can roll up down comparing covers/comments in the book view. If they have custom columns like "Read yes/no" these will be visible to help identify which version to keep. Of course they also have all the existing ability to view formats. Sure we could duplicate most of this functionality into a popup dialog, but that's not a great long term solution imho.

What users don't have in the library view currently is a way of visually identifying duplicate groups. As Charles kindly suggested to me via email one possibility is to use a custom column, which stores a duplicate group number against all the potential candidates as a result of running the duplicate find. Perhaps you could also add a second column giving you some kind of informational message or severity (may not be needed). You would be able to use the tag browser and search capabilities of Calibre to query against/display your duplicate groups. We can also wrap that up in helper menu items in the plugin as well to make it easy to bring back your duplicates.

That imho is the only way for users to have sufficient information available and gui options to make the "hard decisions" about resolving a merge group. The plugin would take responsibility for populating the duplicate groups custom column. With presumably the ability to toggle the custom columns in/out of your library view. And you could launch different types of duplicate searches with the plugin - initially the "exact author, fuzzy title" logic, but eventually other "fuzzier" searches. You can take your time resolving the duplicates across multiple Calibre sessions without rerunning the search since there is no popup dialog. Or you can run it again across a different subsets of records/different matching algorithms etc.

Those are my long winded thoughts for now, comments appreciated as always. There are of course still issues, like clearing the duplicate group custom column values after a merge. And once you start supporting "fuzzier" matching algorithms, you have the issue of repeatedly looking at false positives unless we come up with a way of the user flagging exclusions over time.

[Starson17]

Originally Posted by kiwidude

Awesome - I've got it now.

Update it - I had an error in the code, Kovid fixed part of it, I fixed some more, he improved my code logic, etc., so it's changed a few times.

once you start supporting "fuzzier" matching algorithms, you have the issue of repeatedly looking at false positives unless we come up with a way of the user flagging exclusions over time.

This issue became dominant for me after I'd run my dupes code a few times (and fixed most true dupes). It's why I stopped running it much.

[Starson17]

Originally Posted by kiwidude

So an initial duplicate finder release that reuses the exact same comparison logic automerge has today (exact match author, fuzzy title) will help out all but the last two situations. Which imho is the 90% scenario for users, but I have absolutely no facts to support this so feel free to disagree. It is certainly the easiest use case.

Yes. I wasn't disagreeing with your choice to implement the same logic first, just explaining why we didn't get more aggressive in automerge.

one possibility is to use a custom column, which stores a duplicate group number against all the potential candidates as a result of running the duplicate find.

That sounds like a good approach. If duplicate group numbers are assigned in sequence, you can highlight odd/even duplicate number groups using the highlighting tools. Then the user would easily see the members of each group when sorted by that column.

[kiwidude]

Sorry for the long ramble, it was the closest I had come to trying to organise my thoughts. If I was heading off reservation with them I would much prefer the public tomato throwing for a few posts over days of wasted dev effort...

Originally Posted by Starson17

That sounds like a good approach. If duplicate group numbers are assigned in sequence, you can highlight odd/even duplicate number groups using the highlighting tools. Then the user would easily see the members of each group when sorted by that column.

Yes I must confess to give additional credit to Charles I think he mentioned the same thing and as my post was super long already I didn't . I haven't used the highlighting feature personally as yet so wasn't sure how exactly it fit in but your description sounds great.

[kacir]

Originally Posted by kiwidude

What users don't have in the library view currently is a way of visually identifying duplicate groups. As Charles kindly suggested to me via email one possibility is to use a custom column, which stores a duplicate group number against all the potential candidates as a result of running the duplicate find.

This seems to be the best approach. I am almost sorry it wasn't me that came up with such a clever solution ;-)
This way you do not need to do any UI for merging. User simply runs plugin *unattended* (very important for large libraries ;-) ) and then sorts books by that column. Then the user can merge the duplicates at his/her leisure.

[Starson17]

Originally Posted by kacir

This seems to be the best approach. I am almost sorry it wasn't me that came up with such a clever solution ;-)
This way you do not need to do any UI for merging. User simply runs plugin *unattended* (very important for large libraries ;-) ) and then sorts books by that column. Then the user can merge the duplicates at his/her leisure.

Yes, and after he's run the duplicate finder, and merged or fixed dupes, he could assign "00" as the duplicate group number to flag any duplicates he doesn't want to find later. It would avoid repeatedly looking at false positives. A new run of the dupes finder would rewrite the dupe group column, but ignore any "00" flagged files.