[GUI Plugin] Find Duplicates - Page 3

Starson17 · 05-06-2011, 04:56 PM

Quote:

Originally Posted by kiwidude

The only idea I can recall being suggested for this that is outstanding is the ability to filter the result groups. I don't want to add it unless more than one person requests it though

I don't want to weigh in on how useful that feature might be to others, but it's not one I'd be likely to use.

Stampercam · 05-06-2011, 10:47 PM

I love love love this plugin. You have saved me hours of work. Many thanks.
Cam

wolfelric · 05-08-2011, 12:10 PM

Awesome, I have been needing this for awhile.

geordiejohn · 05-13-2011, 10:03 AM

hello i have downloaded the zip file but can you please tell me how to install in Calibre.
Thank you.

theducks · 05-13-2011, 10:28 AM

Quote:

Originally Posted by geordiejohn

hello i have downloaded the zip file but can you please tell me how to install in Calibre.
Thank you.

Open Preferences
click plugins (icon)
at the very bottom

Try the big button

point the file browser to the downloaded ZIP (do not un-zip)

Noughty · 05-13-2011, 12:10 PM

Quote:

Originally Posted by kiwidude

You will lose any book or author exemptions when you upgrade to this version.

Since you said it is important wanted to ask before updating. I don't know what exemptions are

. If it is only related to this plug in, then I can update with no worry, since I just installed this plug in. But just in case it is something other I wanted to check

kiwidude · 05-13-2011, 12:38 PM

Exemptions are created in this plugin only. They are when you tell the plugin that a specific group of books (or authors, depending on the type of duplicate check you ran) that it presented you with are actually not really duplicates. It means that the next time you run the duplicate check you will not be presented with that particular combination of books appearing together again.

You create exemptions using the menu option "Mark group as exempt" on the plugin menu, or "Mark all groups as exempt".

It becomes particularly useful when you use some of the "fuzzier" algorithms which are more likely to present you with groups of books that actually are not duplicates of each other.

For example an "Ignore Title, Fuzzy Author" search takes the first letter of an authors first name along with their last name and compares those, so that books by "L. Child" and "Lee Child" would appear as a group. It could be that you have another author called "L.J. Child" in your database that would also be in that duplicate group from this algorithm, but in actual fact it is a valid different author. So you would want to resolve any genuine duplicates out of the group (e.g. lets say L. Child and Lee Child were duplicates so you renamed the L. Child book authors to Lee Child). Then you can mark what remains as exempt in future, which would be an exemption of L.J. Child with Lee Child.

Note that exemptions are for that particular combination of book titles or authors, not that they are never considered as possible duplicates at all. If for instance you added a book by an author called Larry Child and ran the same search again of Ignore Title, Fuzzy Author, then (Larry Child, Lee Child) would be presented as one group, and (Larry Child, L.J. Child) would be presented as a second group. Because of the exemption it is not putting all three authors together in the same group.

The change made from 1.0 to v1.1 of this plugin was to change the way exemptions were stored in the configuration file to something far more efficient. However by doing this, it meant that any users of the 1.0 version would lose their exemptions they had already created, as it was too complex to try to migrate them to the new way of storing them. That is what the warning was about.

Hope that makes sense.

garyft · 05-13-2011, 03:12 PM

I installed this and it doesn't seem to give me a button or any way to use it. Just installed on .8.

kiwidude · 05-13-2011, 03:29 PM

Did you add it to any toolbars or menus? If you missed that step, go to Preferences->Toolbar. For more information see the Introduction to Plugins sticky thread.

garyft · 05-13-2011, 05:33 PM

Kiwidude, thanks for the help. That worked great!

Giuseppe Chillem · 05-16-2011, 02:02 PM

Can't Download The Zip file either with Firefox 4 and IE9.

A page full of characters is open.

Giuseppe Chillemi

kiwidude · 05-16-2011, 03:34 PM

@Giuseppe - there must be something going on with your downloads, there is nothing wrong with the zip file on the site. Perhaps anti-virus or something else interfering? Also you could try instlaling it via the plugin updater plugin - assuming you are able to download its zip file of course...

lgladen · 05-17-2011, 01:32 AM

Quote:

Originally Posted by Giuseppe Chillem

Can't Download The Zip file either with Firefox 4 and IE9.

A page full of characters is open.

Giuseppe Chillemi

@Giuseppe -

That means your browser is set to "open" the zip file, instead of saving it - it's trying to "read" the file like a notepad, instead of downloading it to a location you can point calibre to. You might also check the file association settings in your operating system - it might be set wrong for "zip" files.

Narcio · 05-17-2011, 11:28 PM

I have been using the binary compare mode for my rather large library (its a bit of a mess.)

I notice that if you have multiple formats for one book, the plugin seems to flag the book as a duplicate, but it does not provide an easy way to determine which (one, multiple, or all) of the formats of a book are duplicates.

example: Lets say you have two book entries in Calibre, A and B, each with formats x, y, and z. If any one of the formats x, y, or z are binary duplicates in both A and B, it will flag books A and B as dups even if the other formats are not.

I find there is no easy / non tedious way to figure out which of the formats are dups and which aren't. Is there any quick way to figure this out? What I've been doing so far is to look at the file size for each format, but this is time consuming and also not a sure solution.

BTW, thanks for the plugin; I've used it extensively.

kiwidude · 05-18-2011, 04:51 AM

@narcio - yes what you are doing is the only way. I did raise this exact issue on the development thread for this plugin.

My further thought on this was that the problem you are trying to solve now is a merge issue. As what you are wanting to do is merge those book rows together, taking into account that for any binary duplicate formats within that set there is no user intervention required. My idea for a solution for this which I also mentioned on that thread is a new "smart merge" plugin. This would just be another type of merge, but give you a popup dialog detailing the books and formats in the selected rows. So you could double click on formats that are duplicates to view them to decide individually which to keep etc. If there are no duplicate formats within the group, or if the only duplicate formats are binary duplicates, then it will be able to make all the right decisions for you without you rolling up and down the rows trying to figure out which duplicate formats there are, opening them up and trying to remember which format you opened from which row etc.

However that is nothing more than an idea for a plugin at this point, and it has had no serious thought on what it should look like. There is also the question of book metadata, as to whether as a user you should be able to pick and choose fields from that to win. Thinking about this now I think that is unnecessary, particularly given how much better the metadata sourcing is now in 0.8 compared to previous. It could always be added as a later feature with a second tab anyways if it was thought worthwhile.

So the only other problem is the viewing of formats. In a perfect world what I would like to have is a side by side ebook viewer for this scenario. So you could pick a format with duplicates and open up that format to see them next to each other and decide which to keep. However this is unlikely to ever happen, as it would require calibre changes that would only be useful in this one scenario, and still only marginally so, way too much work.

What might be more practical is if there was a command line argument added to the ebook viewer allowing passing of a window title rather than it just being read from the book metadata. So when you opened a pair of duplicate formats, it would pop open the two book format windows but you could see something in the window title to know which is which from the merge dialog. Maybe this ability exists already, I haven't checked.

Sorry to have rambled onto another subject but my point is there isn't a practical alternative I can think of. What the binary duplicate check is telling you is that you have a group of two or more books that need merging because at least one of their formats is a binary duplicate. A menu item to remove all but one of the binary duplicate formats from that group would be a simpler solution but it only addresses one of the issues that you will need to face when merging the group. I would rather we figure out a more generally useful merge plugin approach that has binary comparison as one of it's weapons to help give a more intelligent merge default.

05-17-2011, 11:28 PM	#44
Narcio Junior Member Posts: 2 Karma: 10 Join Date: May 2011 Device: none	I have been using the binary compare mode for my rather large library (its a bit of a mess.) I notice that if you have multiple formats for one book, the plugin seems to flag the book as a duplicate, but it does not provide an easy way to determine which (one, multiple, or all) of the formats of a book are duplicates. example: Lets say you have two book entries in Calibre, A and B, each with formats x, y, and z. If any one of the formats x, y, or z are binary duplicates in both A and B, it will flag books A and B as dups even if the other formats are not. I find there is no easy / non tedious way to figure out which of the formats are dups and which aren't. Is there any quick way to figure this out? What I've been doing so far is to look at the file size for each format, but this is time consuming and also not a sure solution. BTW, thanks for the plugin; I've used it extensively. Last edited by Narcio; 05-17-2011 at 11:36 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] Quality Check	kiwidude	Plugins	1228	Yesterday 08:04 PM
[GUI Plugin] Generate Cover	kiwidude	Plugins	844	01-15-2025 05:06 PM
[GUI Plugin] View Manager	kiwidude	Plugins	415	05-11-2024 04:28 AM
[GUI Plugin] Open With	kiwidude	Plugins	403	04-01-2024 09:39 AM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 01:27 PM

05-06-2011, 10:47 PM	#32
Stampercam Connoisseur Posts: 86 Karma: 19674 Join Date: Jan 2011 Device: ipad, MiGear	I love love love this plugin. You have saved me hours of work. Many thanks. Cam

05-08-2011, 12:10 PM	#33
wolfelric Junior Member Posts: 5 Karma: 10 Join Date: Dec 2008 Device: sony reader 505	Awesome, I have been needing this for awhile.

05-13-2011, 10:03 AM	#34
geordiejohn Zealot Posts: 112 Karma: 861762 Join Date: Dec 2010 Location: London Device: Kindle 3	hello i have downloaded the zip file but can you please tell me how to install in Calibre. Thank you.

05-13-2011, 12:38 PM	#37
kiwidude Calibre Plugins Developer Posts: 4,694 Karma: 2162246 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Exemptions are created in this plugin only. They are when you tell the plugin that a specific group of books (or authors, depending on the type of duplicate check you ran) that it presented you with are actually not really duplicates. It means that the next time you run the duplicate check you will not be presented with that particular combination of books appearing together again. You create exemptions using the menu option "Mark group as exempt" on the plugin menu, or "Mark all groups as exempt". It becomes particularly useful when you use some of the "fuzzier" algorithms which are more likely to present you with groups of books that actually are not duplicates of each other. For example an "Ignore Title, Fuzzy Author" search takes the first letter of an authors first name along with their last name and compares those, so that books by "L. Child" and "Lee Child" would appear as a group. It could be that you have another author called "L.J. Child" in your database that would also be in that duplicate group from this algorithm, but in actual fact it is a valid different author. So you would want to resolve any genuine duplicates out of the group (e.g. lets say L. Child and Lee Child were duplicates so you renamed the L. Child book authors to Lee Child). Then you can mark what remains as exempt in future, which would be an exemption of L.J. Child with Lee Child. Note that exemptions are for that particular combination of book titles or authors, not that they are never considered as possible duplicates at all. If for instance you added a book by an author called Larry Child and ran the same search again of Ignore Title, Fuzzy Author, then (Larry Child, Lee Child) would be presented as one group, and (Larry Child, L.J. Child) would be presented as a second group. Because of the exemption it is not putting all three authors together in the same group. The change made from 1.0 to v1.1 of this plugin was to change the way exemptions were stored in the configuration file to something far more efficient. However by doing this, it meant that any users of the 1.0 version would lose their exemptions they had already created, as it was too complex to try to migrate them to the new way of storing them. That is what the warning was about. Hope that makes sense.

05-13-2011, 03:12 PM	#38
garyft Junior Member Posts: 4 Karma: 10 Join Date: May 2011 Device: Kindle	I installed this and it doesn't seem to give me a button or any way to use it. Just installed on .8.

05-13-2011, 03:29 PM	#39
kiwidude Calibre Plugins Developer Posts: 4,694 Karma: 2162246 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Did you add it to any toolbars or menus? If you missed that step, go to Preferences->Toolbar. For more information see the Introduction to Plugins sticky thread.

05-13-2011, 05:33 PM	#40
garyft Junior Member Posts: 4 Karma: 10 Join Date: May 2011 Device: Kindle	Kiwidude, thanks for the help. That worked great!

05-16-2011, 02:02 PM	#41
Giuseppe Chillem Groupie Posts: 178 Karma: 134 Join Date: May 2010 Device: IREX DR1000	Can't Download The Zip file either with Firefox 4 and IE9. A page full of characters is open. Giuseppe Chillemi

05-16-2011, 03:34 PM	#42
kiwidude Calibre Plugins Developer Posts: 4,694 Karma: 2162246 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@Giuseppe - there must be something going on with your downloads, there is nothing wrong with the zip file on the site. Perhaps anti-virus or something else interfering? Also you could try instlaling it via the plugin updater plugin - assuming you are able to download its zip file of course...

05-18-2011, 04:51 AM	#45
kiwidude Calibre Plugins Developer Posts: 4,694 Karma: 2162246 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@narcio - yes what you are doing is the only way. I did raise this exact issue on the development thread for this plugin. My further thought on this was that the problem you are trying to solve now is a merge issue. As what you are wanting to do is merge those book rows together, taking into account that for any binary duplicate formats within that set there is no user intervention required. My idea for a solution for this which I also mentioned on that thread is a new "smart merge" plugin. This would just be another type of merge, but give you a popup dialog detailing the books and formats in the selected rows. So you could double click on formats that are duplicates to view them to decide individually which to keep etc. If there are no duplicate formats within the group, or if the only duplicate formats are binary duplicates, then it will be able to make all the right decisions for you without you rolling up and down the rows trying to figure out which duplicate formats there are, opening them up and trying to remember which format you opened from which row etc. However that is nothing more than an idea for a plugin at this point, and it has had no serious thought on what it should look like. There is also the question of book metadata, as to whether as a user you should be able to pick and choose fields from that to win. Thinking about this now I think that is unnecessary, particularly given how much better the metadata sourcing is now in 0.8 compared to previous. It could always be added as a later feature with a second tab anyways if it was thought worthwhile. So the only other problem is the viewing of formats. In a perfect world what I would like to have is a side by side ebook viewer for this scenario. So you could pick a format with duplicates and open up that format to see them next to each other and decide which to keep. However this is unlikely to ever happen, as it would require calibre changes that would only be useful in this one scenario, and still only marginally so, way too much work. What might be more practical is if there was a command line argument added to the ebook viewer allowing passing of a window title rather than it just being read from the book metadata. So when you opened a pair of duplicate formats, it would pop open the two book format windows but you could see something in the window title to know which is which from the merge dialog. Maybe this ability exists already, I haven't checked. Sorry to have rambled onto another subject but my point is there isn't a practical alternative I can think of. What the binary duplicate check is telling you is that you have a group of two or more books that need merging because at least one of their formats is a binary duplicate. A menu item to remove all but one of the binary duplicate formats from that group would be a simpler solution but it only addresses one of the issues that you will need to face when merging the group. I would rather we figure out a more generally useful merge plugin approach that has binary comparison as one of it's weapons to help give a more intelligent merge default.

Advert

Advert