[GUI Plugin] Zotero Metadata Importer

[DaltonST]

You probably have noticed that there have been a huge number of changes to Zotero as of the newly released version 5.0.

I have been testing this plug-in, ZMI, using test data created by and exported from an entirely fresh installation of Zotero 5.0.

So far, the only significant issue that I have personally detected is due to technical changes in certain of Zotero's SQLite database tables that ZMI reads in order to compare what "books" are in Calibre versus Zotero. Therefore, as of the upcoming new version of ZMI that I will upload in the next few days, the "ZMI:Comparison" Tab will not work unless you are using Zotero 5. You will get a ZMI error message stating that you must upgrade to 5.

All of the other functions should continue to work just as before, since they are based on the Zotero "Collections" exported .csv file, and not the Zotero storage SQLite database.

If you find any problems, I will need a .zip file of a "storage" folder containing your attachments (.pdf files) plus the .csv file that was exported by you for the items in that storage folder so that I will be able to recreate your issues. In other words, I will need "real data" from "real users" to test with, along with a detailed explanation of exactly what the issue is.


Thanks.



DaltonST

[DaltonST]

Version 1.0.42 - Technical changes for Zotero 5.0 Compatibility, plus tweaks related to Calibre 3.0 changes.

All development and testing of ZMI is now performed on Calibre 3+ and Zotero 5+. If you have problems, and are not on the latest versions of both Calibre and Zotero, helping you will be problematic.



DaltonST

[DaltonST]

The next release of ZMI will have a totally redesigned Comparison Tab. It will compare Zotero Items to Calibre Books instead of comparing Zotero File Attachments to Calibre Books. Also, non-attachments (i.e., Calibre Books that are not .pdf files in Zotero) will have their Zotero Collection updated in Calibre, just like file attachments have had. See the attached. The next ZMI release will be uploaded within a couple of days.


DaltonST

[DaltonST]

Version 1.0.45 - 2017-07-25 Fixed item "title".
Version 1.0.44 - 2017-07-25 Comparison Tab enhancements.

The Comparison Tab has been enhanced to provide more Zotero detail for any items missing in Calibre.

To continue to use the Comparison Tab, you must have upgraded to Zotero 5.0+ due to Zotero database changes that impact ZMI.



DaltonST

[DaltonST]

Version 1.0.48 - 2017-07-28 Fixed regression error in mark/select new ZMI book interaction.



DaltonST

[DaltonST]

The next release of ZMI that will be uploaded within a few days will allow you to export Calibre metadata to an RIS file, and then import that RIS file into Zotero. Zotero will map that RIS data to Zotero's data fields for a "book".

It is up to you to decide what Calibre metadata is exported to the RIS file. Zotero totally controls the mapping of an RIS file to itself. ZMI offers all available Zotero mappings for a "book" for your use. It is up to you to decide what Calibre metadata is mapped into the Zotero fields that Zotero has predefined.

ZMI must work within the constraints set by Zotero in its RIS.js import translator.


See the attached.


Note that you must use standard Zotero functionality to "attach" or "link" a physical Calibre ebook file to your Zotero item. ZMI just handles that book's metadata.


DaltonST

[DaltonST]

Version 1.0.49 - 2017-08-04 New ZMI Tab: Export RIS File. Export Calibre metadata as .ris file designed specifically for import into Zotero.


Refer to the respective attachment to the ZMI Original Post for details.



DaltonST

[DaltonST]

Version 1.0.50 - 2017-09-22 Calibre 3.8 now required. Update Calibre prior to updating ZMI. Functionality changes for Calibre 3.8.0 compatibility related to setting the 'covers' for newly imported Zotero 'books'.

ZMI previously added a newly imported Zotero "book" cover to the Calibre metadata for each book because Calibre did not have a user-friendly way to automatically add the covers en masse. As of Calibre version 3.8, Calibre does. At the bottom of Edit Metadata in Bulk, there is a new option to do so. ZMI has been changed to notify you to use this new Calibre functionality, and its old workaround for the lack of this functionality in older versions of Calibre has been removed.

Be sure to update Calibre to 3.8+ prior to updating ZMI.



DaltonST

[brokengillou]

Hi,
I tried ZMI (all softs updated) with a distinct new Calibre library, tried to import some books from a Zotero collection of empty books (only library links references from sudoc).
No empty book created in Calibre.
Is it the normal behavior?
If yes would it be possible to offer this option as it is already possible to create empty books in Calibre.
Thanks by advance.
Gilles

[DaltonST]

brokengillou:

There is already a ZMI option in the ZMI:Options Tab called "Create an 'empty book' for CSV items with no file attachments?".

There is also already a ZMI option called "import .txt and .text attachments too?".

If these do not work for you, then you need to attach your .csv export file so I can look at your real data, because otherwise there is no way for me to know exactly what it is that you are talking about.

Added: if you export your library to a .csv from Zotero, open that .csv with a spreadsheet application, and go far to the right where the column "File Attachments " is found (normally column AL or so), you should see the exact file related to your question. If the "File Attachments" column is empty (blank), then there is nothing that ZMI can do other than create an "empty book" if that ZMI option is checked "yes".


Important: if you want to import a "web-page,complete" into ZMI, you will have to first bundle all of its many little component files together using an existing Zotero plug-in into some single package format that ZMI can pre-process before adding it to Calibre with all of the 26 ZMI custom columns that go with a Zotero file attachment. Example: a .zip or .rar file. ZMI processes single file attachments only (pdf, txt, text, epub, mobi, docx, odt, pptx, xls, zip, rar, etc. etc.).



DaltonST

[brokengillou]

Hi Dalton,

Sorry to confirm i used the option "Create an empty book", could you have a look on this csv:
https://e-nautia.com/brokengillou/di...ZMI/Mashup.csv
As i select my csv in either auto or manual mode no book list appears to apply further operations on. I expected to get to this first step then trying to assign sudoc number in link to identifiers as sudoc:########### (also isbn).
Thanks,
Gilles
NB: I found this tool relative to SUDOC:
https://www.npmjs.com/package/sudoc

[DaltonST]

This is why (extracted from your .csv file):

File Attachments
C:\Users\leguillou-g\Zotero\storage\DJTF7VFA\137469829.html; ;
C:\Users\leguillou-g\Zotero\storage\EN7K9BD9\151081263.html; ;
C:\Users\leguillou-g\Zotero\storage\X5RTK92Z\094641773.html; ;
C:\Users\leguillou-g\Zotero\storage\726HJPIG\169637190.html; ;
C:\Users\leguillou-g\Zotero\storage\XA5AFUD3\07075649X.html; ;
C:\Users\leguillou-g\Zotero\storage\I6EZFCHV\076181499.html; ;
C:\Users\leguillou-g\Zotero\storage\92E43E93\068551894.html; ;
C:\Users\leguillou-g\Zotero\storage\Q8XQRZNR\060756632.html; ;
C:\Users\leguillou-g\Zotero\storage\HK6QSH8I\060972882.html; ;
C:\Users\leguillou-g\Zotero\storage\CQHQ3K78\159350379.html; ;
C:\Users\leguillou-g\Zotero\storage\8KHGADD9\147110335.html; ;
C:\Users\leguillou-g\Zotero\storage\U7R2EJZH\070838658.html; ;
C:\Users\leguillou-g\Zotero\storage\Z955FTFG\116741872.html; ;
C:\Users\leguillou-g\Zotero\storage\IBZFH4H6\165792078.html; ;
C:\Users\leguillou-g\Zotero\storage\D5TTIC5X\185273734.html; ;
C:\Users\leguillou-g\Zotero\storage\G2PKQH7R\060447184.html; ;

As my prior post mentioned, ZMI ignores Zotero HTML "web-pages,complete" because they are not a single file, but rather a directory filled with collection of many different little files, such as with extensions .js and .png and .html.

If you want to import a "web-page,complete" into ZMI, you will have to first bundle all of its many little component files together using an existing Zotero plug-in into some single package format that ZMI can pre-process before adding it to Calibre with all of the 26 ZMI custom columns that go with a Zotero file attachment. Example: a .zip or .rar file. ZMI processes single file attachments only (pdf, txt, text, epub, mobi, docx, odt, pptx, xls, zip, rar, etc. etc.).

I believe that Zotero offers a Zotero plug-in that will package your "web-pages,complete" folders into a .zip or a .pdf so that ZMI can import it.

The URL for all Zotero plug-ins is: https://www.zotero.org/support/plugins.

I suggest: http://zotfile.com/ since ZotFile seems to be the Zotero "go-to" plug-in for file attachment manipulation. It probably can convert each of your html folders into discrete .pdf files, which would be perfect for ZMI to import.

Added: as a quick-and-dirty (unofficial) workaround just to get your Zotero Item Metadata into Calibre without any file attachments, simply edit your Zotero export .csv file to delete any of the .html file attachment paths shown above. ZMI can then create an empty book for them, but with all of the Zotero metadata found in the .csv file. You will not have the original web page, of course. Later, you could drag-and-drop any "real" book format that you can somehow create for that web page onto the Calibre empty book to add that "real" book format.


DaltonST

[brokengillou]

Ok! First goal achieved.

Thank you for your explanations, could suggest a log or message box to get through this. I used your standard metadata transfer with success except for language (however in zotero...).
Main concern now, if i show identifiers column i see some zkey_file, zkey, zmi attributes, is this information convenient in this field?
Anyway, next i had to modify content of "Link Attachements" from usual url to idlib:###...### to zotero_extra as the field "Link Attachements" is not imported by ZMI.
At this point i really believed ZMI might have done the job with #zotero_extra:identifiers but no way (#zotera_extra:#temp is working but is not very helpful). Just to clarify, next i have custom fields for isbn, sudoc, worldcat...
What can you suggest as a work arround?

Thanks!

Gilles

PS: Zotfile seems to be incompatible with current Zotero version.

https://addons.mozilla.org/fr/firefox/addon/zotfile/
error: L'extension "ZotFile" ne peut pas être installée. Elle est peut-être incompatible avec cette version de Zotero Standalone
Firefox: Ce module n'a pas pu être installé car il n'a pas pu être vérifié.

[DaltonST]

if i show identifiers column i see some zkey_file, zkey, zmi attributes, is this information convenient in this field?

Those z* Identifiers are critical to the workings of ZMI. Among other things, they provide an unambiguous audit trail from Calibre back to Zotero. Do not delete or change them. If you do, ZMI will not know what Zotero Items are already in Calibre and will add them again, or what newly created ZMI "books" are still in-process (incomplete) and should continue to be processed by ZMI. Other undesirable things can happen too.

n.b. The English word "convenient" is not used in this context. You were looking for the word "appropriate" (approprié).

#zotero_extra:identifiers but no way (#zotera_extra:#temp is working but is not very helpful)

The pushbutton called "Copy Miscellaneous ZColumns to Calibre Custom Columns [Selected Books] does just that: it copies imported Zotero metadata to Custom Columns. The ToolTips for that pushbutton explain that. The Calibre metadata column "Identifiers" is a standard column, not a Custom Column.

The only standard columns that the ZMI pushbutton called "Copy Miscellaneous ZColumns to Standard Calibre Columns [Selected Books]" copies from the appropriate ZColumns are Tags, Publisher, and Publication Date. That is because the Zotero definitions of those fields exactly match the Calibre definition of those fields. The ToolTips for that pushbutton explain that.

If you wish to copy ZColumns to arbitrary and random standard columns, you should use the Calibre Bulk Edit-Search and Replace dialog. This should be of use: https://manual.calibre-ebook.com/ .

i have custom fields for isbn, sudoc, worldcat...

If you look at this thread's Original Post, there is an attachment showing all 26 Calibre Custom Columns that are mapped from the Zotero export .csv file. You can also view them in Calibre > Preferences > Add Your Own Columns, and also see whether or not they are "hidden columns" in Calibre. You will see that #zotero_isbn is one of them. Your mission is to put your ISBN in Zotero into the Zotero ISBN field so it will automatically be mapped into #zotero_isbn. You can choose to "unhide" #zotero_isbn.


ZMI is a Calibre plug-in, not a Zotero plug-in. What Zotero metadata gets written to which export .csv file column is totally a Zotero developer decision. All you can control is putting Zotero metadata, such as your sudoc and worldcat numbers, into Zotero metadata fields such that they get mapped into the Zotero export .csv file so that they then get mapped to a Calibre #zotero_xxxxx Custom Column.

I noticed from your Zotero export .csv file that your Zotero Item Type is always "book". There is only a subset of possible Zotero metadata fields available for "book" as compared to others Item Types. For example, ZMI has a Calibre Custom Column named "#zotero_issn". ISSN is available in Zotero for periodicals, not "books". That means that you need to determine how to manipulate Zotero's Item Type to get your Zotero metadata for sudoc and worldcat into some Zotero metadata column so that it gets written by Zotero to its export .csv file within one of the 26 Zotero export .csv file columns that ZMI will import into Calibre.

PS: Zotfile seems to be incompatible with current Zotero version.

ZotFile is compatible with Zotero 5, which is the latest version. Full use of ZMI requires Zotero 5, since the SQLite database that Zotero uses had changes in some field names in Version 5.

Spoiler:

Per https://github.com/jlegewie/zotfile For Zotero 5, first download the extension file (follow the download link above, click on the .xpi file for the most recent release). Now start Zotero 5 and go to "Tool -> Add-ons -> Tools for all Add-ons (the small, drop-down wheel in the top right corner) -> Install Add-on From File" and select the downloaded .xpi file. For Zotero 4.x, the process is different for Zotero for Firfox and Zotero Standalone. For Zotero 4.x Firefox, go to the Mozilla Add-Ons page and follow the instructions. For Zotero 4.x Standalone, use the same steps as for Zotero 5 but download the .xpi file from here

DaltonST

[DaltonST]

Version 1.0.53 - 2017-10-22 New ZMI option to create an empty book if the sole file attachment is html.

The ToolTips for both the new option and its related prerequisite option are shown in the attached image.



DaltonST