calibredb 'merge'

[kjdavies]

Is there a way to have calibredb use the 'merge' action I use via the GUI? This not the same as adding a format.

I am cleaning up a few libraries I have integrated. I have a reliable identifier I can use to identify duplicates (CRC,extension,filesize) but other associated information such tags and tag-like fields.. These are bitwise-identical files, but I want to aggregate the supplementary information.

What I'd like to do is write a script that will examine all sets of titles with the same unique key, and where the titles and authors match, merge them into a single entry. (I'll want to examine where these things differ to see which ones I prefer, so I'll still need to handle these ones manually.)

I could make my script a little more sophisticated, of course, and handle the aggregation myself via set_metadata... but if it's already built-in, I'd rather do that. It's less work for me, and much more importantly, less prone to error.

[theducks]

Does calibredb --automerge, -m work?

If books with similar titles and authors are found, merge the incoming formats (files) automatically into existing book records. A value of "ignore" means duplicate formats are discarded. A value of "overwrite" means duplicate formats in the library are overwritten with the newly added files. A value of "new_record" means duplicate formats are placed into a new book record.

the number in the containing Folder is the books {ID} if that helps

[kjdavies]

Originally Posted by theducks

Does calibredb --automerge, -m work?

If books with similar titles and authors are found, merge the incoming formats (files) automatically into existing book records. A value of "ignore" means duplicate formats are discarded. A value of "overwrite" means duplicate formats in the library are overwritten with the newly added files. A value of "new_record" means duplicate formats are placed into a new book record.

the number in the containing Folder is the books {ID} if that helps

Not what I'm looking for. The books are already in the library, I'm confident they are the same book (bitwise identical), but I want to merge the the metadata.

I'm thinking of a command something like

calibredb merge id1 id2 id3

If I've got a '#notes' custom column, and the three titles with IDs id1, id2, id3 have #notes values of 'Note 1', 'Note 2', and 'Note 3', when I merge via the GUI I get a #notes value of 'Notes1\n\nNotes2\n\nNotes 3'. Taglike fields are similarly aggregated and the union stored in the result.

[capink]

Originally Posted by kjdavies

Is there a way to have calibredb use the 'merge' action I use via the GUI? This not the same as adding a format.

I am cleaning up a few libraries I have integrated. I have a reliable identifier I can use to identify duplicates (CRC,extension,filesize) but other associated information such tags and tag-like fields.. These are bitwise-identical files, but I want to aggregate the supplementary information.

What I'd like to do is write a script that will examine all sets of titles with the same unique key, and where the titles and authors match, merge them into a single entry. (I'll want to examine where these things differ to see which ones I prefer, so I'll still need to handle these ones manually.)

I could make my script a little more sophisticated, of course, and handle the aggregation myself via set_metadata... but if it's already built-in, I'd rather do that. It's less work for me, and much more importantly, less prone to error.

A combination of Find Duplicates and Action Chains plugin can help you achieve what you want:

1. Use the Find Duplicates plugin to group the identical books.
2. Using Action Chains plugin, you can use this custom module to auto-merge the book records of the same duplicate group. The will be merged into the first book of each group.

Notes:

• To add the custom module, go to Action Chains > Manage Module > add
• This is a destructive action after which you will not be able to restore your merged records, make sure you backup you library before you proceed. You are ultimately responsible for any damage that might happen.

[kjdavies]

Originally Posted by capink

A combination of Find Duplicates and Action Chains plugin can help you achieve what you want:

1. Use the Find Duplicates plugin to group the identical books.
2. Using Action Chains plugin, you can use this custom module to auto-merge the book records of the same duplicate group. The will be merged into the first book of each group.

Notes:

• To add the custom module, go to Action Chains > Manage Module > add
• This is a destructive action after which you will not be able to restore your merged records, make sure you backup you library before you proceed. You are ultimately responsible for any damage that might happen.

This sounds pretty close to what I want, so thank you for that. I've been meaning to learn action chains.

Can it be smart enough to also require that the titles match? Within a group I don't always have the titles match and would like to be able to pick which ones I want.

(Yes, I realize I might be more or less unique in my need here...)

[capink]

Originally Posted by kjdavies

This sounds pretty close to what I want, so thank you for that. I've been meaning to learn action chains.

Can it be smart enough to also require that the titles match? Within a group I don't always have the titles match and would like to be able to pick which ones I want.

(Yes, I realize I might be more or less unique in my need here...)

The solution to this is to choose the files you do not want to be merged, apply a tag to them, let say "exempt_unique", exit Find Duplicates view, apply a virtual library with the search

not tag:"=exempt_unique"

and then re-run Find Duplicates while in the virtual library, this way none of these books would show in the duplicates results.

[kjdavies]

Originally Posted by capink

The solution to this is to choose the files you do not want to be merged, apply a tag to them, let say "exempt_unique", exit Find Duplicates view, apply a virtual library with the search tag:"=exempt_unique", and then re-run Find Duplicates while in the virtual library, this way none of these books would show in the duplicates results.

It comes to me that it might not be an important detail. This is just deduplicating the bitwise-identical files. I still have to review the results and bring them up to my standard. Being able to 'pick' the preferred title perhaps doesn't gain me much in that scenario.

[capink]

Originally Posted by kjdavies

It comes to me that it might not be an important detail. This is just deduplicating the bitwise-identical files. I still have to review the results and bring them up to my standard. Being able to 'pick' the preferred title perhaps doesn't gain me much in that scenario.

OK, Just in case you decide to use what I posted in my previous post, the correct search would be:

not tag:"=exempt_unique"

[kovidgoyal]

No there's no CLI interface for merging metadata.

[kjdavies]

Originally Posted by kovidgoyal

No there's no CLI interface for merging metadata.

Thanks Kovid, I was pretty sure. The documentation didn't mention it.

Is it worth submitting an enhancement request? I'm thinking the semantics would be something like

calibredb merge [options] id1 id2 [... idN]

Options might include

• --library-path, of course
• -h,--help, obviously
• --version
• setting/adding tag(s) -- user-specified would be nicest, but even a hardcoded 'calibredb-merged-this' would be great
• setting other metadata (tag could be enough, though)
• 'safe' (will refuse if it tries to merge two titles with the same format -- my use case specifically is to merge titles with the same formats, I'm deduping, but I can see value to this)

[kovidgoyal]

Not unless you are willing to implement it as well

[kjdavies]

Originally Posted by kovidgoyal

Not unless you are willing to implement it as well

If I had the knowledge. Python is probably one of my weakest languages.

I mean, LISP is lower (I've done some work with Schema via DSSSL... 20+ years ago) and I don't even admit to having seen COBOL, but of languages in common use today, Python is probably one of my weakest.

[chaley]

@kovid: I looked at the merge_metadata() method in the edit_metadata action. It is very self contained. I copied it to a python template and ran it with almost no changes. Metadata from my test books was merged. Of course it doesn't do anything with formats and doesn't delete any books.

If I move it from the GUI to ebook.metadata.book then it could be used in scripts executed with 'calibre-debug -e' as well as be used by some eventual command line utility (that I am not volunteering to write). A stub must remain in edit_metadata to pass the db to the moved method.

I could do the same thing with merge_data_files(), but I don't think it is necessary. The only thing is does is call the db api merge_extra_files() method.

Is this OK with you?

[ownedbycats]

Originally Posted by kjdavies

but of languages in common use today, Python is probably one of my weakest.

Just for you:

Spoiler:

[kovidgoyal]

@charles: Yes that's fine.