[GUI Plugin] Zotero Metadata Importer

[DaltonST]

Version 1.0.75 - 2022-04-06 Enhanced the Comparison Tab Feature "Show Full Listing [All Books]"

See the attached image.

DaltonST

[DaltonST]

Version 1.0.76 - 2022-04-14 Miscellany. Qt6 compatibility. Minimum Calibre version now 6.0.0.

[snow-raven]

I've had some issues with imports recently that are causing troubles ranging from significant to annoying. From most to least concerning:

- My last two imports have resulted in only some of the Zotero records creating Calibre records, although I have confirmed that they are not currently in my Calibre library. There are at least a few of the missing records that are new enough that I'm at least 90% confident I've never said "no" to importing them with the plugin before.

In tandem with the last issue that I'll mention below, I often have to relaunch Calibre to finish my ZMI import. With these two incidents, both times after restarting it told me there were still missing records from the same CSV, only to find after agreeing to import them that they were duplicates.

I can share a comparison of Zotero record CSV and my Calibre library, if that would be helpful for troubleshooting.

- In these two recent imports, URLs did not copy to the Calibre ID field for any records. This used to happen without me taking any special steps; is it no longer part of the standard manual workflow? I'm able to do it with JobSpy, but I have to fight with the CSV file each time to get all the single quotes formatted in the correct places...

- For possibly about 6 months or so (sorry, I've lost track of exactly when it started), the imported records have not shown up after they are added and the "ZMI" ID does not show in the identifiers list. If the library refreshes or I relaunch Calibre, they appear. I have not been able to locate a way to manually refresh the library, so I typically just relaunch and pick up the ZMI process again. It has been only a minor annoyance until encountering the first issue described above, and now it creates a few duplicate records when I restart ZMI. (And then they don't show in the total "ZMI" ID records, so I relaunch Calibre and restart ZMI, which then prompts me to make a few more duplicate records...).

Thanks as always for continuing to support my favorite Calibre plugin!

[DaltonST]

Your narrative has few facts that are helpful, but I suspect that some of your issues are self-inflicted. Do this:

[1] Upgrade Calibre Desktop (not portable) to 6.7.1. Not optional.

[2] Upgrade ZMI to the latest version for Calibre 6+, which is 1.0.76. Not optional.

[3] Upgrade Zotero Desktop to 6.0.15. Zotero > Help > Check for updates. Not optional. ZMI requires the desktop version, and not the cloud version or a network version. The Zotero Storage folder on your hard drive that was created by Zotero itself is what ZMI uses. Never the cloud version. Never a network location.

[4] Run Calibre for your ZMI Library. Library > Library Maintenance > Check Library. Say "yes" for everything, and fully repair your corrupted metadata and files.

[5] Make a careful manual backup of you metadata.db. Make another full backup of your Calibre Library, including the "books". There is a standard Calibre menu option for doing so.

[6] Run Calibre in full debug mode using " calibre-debug -g " so you can watch in real-time what happens within Calibre and ZMI while ZMI imports a Zotero CSV export file.

[7] Open the ZMI window.

[8] Read the ToolTips for ZMI by hovering your mouse over the various widgets and options. Save any changes, and ZMI will restart.

[9] Create a new Zotero CSV Export file for a single Zotero item that you have previously assigned to a Collection for ease of comparing Zotero to Calibre using the "ZMI: Comparison" feature. Do not make any manual changes to it in Calc/Excel. I suspect you have been making manual edits that have corrupted the CSV file. ZMI requires "double-quoted" delimiters, and a format of UTF-8. FYI, Calc has an "edit filters" option when you save a file: UTF-8, double-quoted delimiters.

[10] Exit from Zotero.

[11] While still in Calibre full debug mode, use the "Auto-Step" option for ZMI, while clicking the "Select single CSV export file" button. Select the CSV file created directly from Zotero in a previous step.

[12] Watch what happens both in ZMI, and then on the debug screen. Note that you can "select all" of the text in debug mode, copy it to the clipboard, and paste it into a .txt document to save it and upload it to this thread so I can see it.

[13] After [12], there is still much more to do. ZMI creates an identifier for "new" books, so click the button "show incomplete books". Calibre will then show in the search bar: identifiers:"=zmi:new" .

It also creates Calibre Identifiers for the Zotero ZKEY, ZKEYFILE, and ZCOLLECTION. The Identifier ZKEY is what is used to drive ZMI. If Calibre Identifier ZKEY is missing from the Calibre Metadata, then that item is "broken" and will not exist as far as ZMI is concerned.

[14] Select all shown books so you can continue updating their Zotero metadata in Calibre using ZMI.

[15] In the middle of the ZMI window for importing from Zotero, click the pushbuttons for "Update Calibre Metadata using Zotero Metadata".

When you get pop-up messages from ZMI, copy them to the clipboard and save them to a .txt file for future reference.

[16] Do NOT archive/delete the CSV file "yet".

[17] Click the "Restart ZMI" buttom at the bottom. After ZMI restarts, click the "ZMI: Comparison" Tab. Update the Zotero DB Directory to where it belongs on your hard drive (not network or the cloud). Click "Zotero Collection" to choose the Collection you want to compare. Hopefully, you assigned a Collection to the Zotero Item that you previously exported to a CSV file and imported to Calibre with ZMI.

[18] Click "Compare Zotero Collection Items to Calibre Books". Check your new Zotero item to see that it is now in Calibre. ZMI updates the Zotero Collection with the Identifier ZCOLLECTION whenever you click the "compare" pushbutton in the "ZMI: Comparison" Tab.


If that single test item works properly, continue doing the same thing.

If the debug log shows a failure during import of the CSV, that is the root cause of your issues. Post the CSV export file with the single Zotero item, the full debug log (saved to a .txt file), and the ZMI messages that you saved to another .txt file to this thread so I can peruse them.



DaltonST

[snow-raven]

Thank you, Dalton.

-I meant to include that I did verify that I am using Calibre Desktop 6.7.1, ZMI 1.0.75, and Zotero 6.0.15. I checked for ZMI updates both before posting and again just now, but I do not see a 1.0.76 available and the plugin status reads "Latest version installed". Could you confirm that I should be updating ZMI through the regular Calibre plugin manager?

-I do use the ZotFile plugin to store attachments in a wired network location. However, these file paths appear correctly in the Zotero export and the connection is present at the time of sync. I have not had any issues with this in the past, either in Zotero or ZMI. Local storage is not an option due to storage volumes and other logistics. I understand that this might risk an attachment not connecting, but would it prevent some records for even appearing in the import?

- I appreciate the rest of your comprehensive list of troubleshooting recommendations and will work on testing those before reporting back.

[DaltonST]

Version 1.0.76 - 2022-04-14 Miscellany. Minimum Calibre version now 5.99.8.

Get it from the original post.

DaltonST

[DaltonST]

Originally Posted by snow-raven

-I do use the ZotFile plugin to store attachments in a wired network location. However, these file paths appear correctly in the Zotero export and the connection is present at the time of sync. I have not had any issues with this in the past, either in Zotero or ZMI. Local storage is not an option due to storage volumes and other logistics. I understand that this might risk an attachment not connecting, but would it prevent some records for even appearing in the import?

Full Calibre Debug mode should answer your question as to whether the ZotFile Path works in ZMI.

I have only standard Zotero installed on my Calibre ZMI test environment of Windows 11 using local SSDs.



DaltonST

[snow-raven]

DaltonST,

My apologies for the delay, but I finally was able to follow your troubleshooting protocol and have a preliminary example for my second problem (no URL). As you requested, I performed a single-file collection comparison, import, and then compared again to successfully import the item and update the metadata. However, the URL does not appear in the Calibre item identifiers, although it does appear in the #zotero_url field.

I am attaching the Zotero collection export (single.csv), the debug session (ZMIDebug.txt), and a Calibre export of the item's metadata (Calibre_IM224CLR.csv). None of these have been edited.

(The only Zotero exports I have edited in the past were to update the URLs to a format that JobSpy+ would recognize to add the missing URLs in Calibre)

Regarding problem # 1, There were 33 files that did not import properly in my last import. I will troubleshoot that separately and update in another post.

Thank you,
snow-raven

ZMIDebug.txt
single.csv.txt
Calibre_IM224CLR.csv.txt

[snow-raven]

Regarding the 33 files that previously failed to import - I separated those into a new collection and tried importing them with a new debug session. As shown in the attached, all 33 were not found in Calibre prior to import and all 33 show as having no content and no path. However, two imported - one a PDF report and the second a website with no attachments. The remainder were an assortment of books, reports, and statutes with PDF attachments. I can identify no obvious difference between those that imported and those that did not. All 33 failed during the first attempt a few weeks ago.

33_ZMIDebug.txt
33_not_found.csv.txt
33_ZMI_Comparison.txt

I tried a smaller sample of 11 of the reports that would not import and simply failed to complete the auto-step process. Calibre stops progressing and does not freeze. The debug shows that all of the rows are located, but seems to stop right before the temp file creation.

11_ZMIDebug.txt
11_ZMI_Comparison.txt
11_not_found_redcd.csv.txt

Please let me know if there are other troubleshooting data I can share.

[DaltonST]

29 Nov 2022
Initial review of your 3 zmidebug.txt files:


[1] db.add_books complete for new book id: 3706 zkey_file: R <<<=== Invalid Format for a ZKEY (Zotero KEY in its storage.db)


[11] I believe that the aborted execution was caused by random file attachment path punctuation characters, such as a semi-colon, ";", as in: Dufour - 1980 - Effects of Noise on Wildlife and Other Animals; Re2.pdf


[33] db.add_books complete for new book id: 3707 zkey_file: DOWL HKM <<<=== Invalid Format for a ZKEY (Zotero KEY in its storage.db) However, since book 3707 had no file attachment, it didn't matter.

The two (2) books that were successfully added did NOT have a file attachment path using: Y:\\
The path for the attachment is a network path: Y:\\

zotero_csv_list: ['A2EZLLBK', 'report', '2019', 'USDOT', 'Anton Larsen Bay Road Extension; Kodiak Island Borough, Alaska', '', '', '', '', 'https://highways.dot.gov/sites/fhwa.dot.gov/files/docs/federal-lands/projects/31236/albre_public-ea_vol-i_dec2019signed.pdf', '', '2019-12-17', '2022-10-17 00:20:34', '2022-10-18 00:11:43', '2022-10-17 00:20:12', '97', '', '', '', '', '', '', '', '', '', '', 'U.S. Department of Transportation, Federal Highway Administration, Western Federal Lands Highway Division', 'Kodiak, AK', 'en', '', 'Environmental Assessment', '', '', '', '', '', '', 'Y:\\HS&E\\Permitting\\Zotero_References\\USDOT\\U SDOT - 2019 - Anton Larsen Bay Road Extension; Kodiak Island Bor.pdf', '', 'EA; NEPA; FHWA; USDOT; road; Kodiak; public; AFLHD', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', 'AK TR OUZI 2017(1)', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']

Initial thoughts:

A. The use of a non-standard storage location for your Zotero file attachments is causing these problems: Y:\\........\\.......\\....

B. Refer to the attached diagram of how standard Zotero is supposed to work, and thus how ZMI does work.

C. You might have to use some technical legerdemain to make your networked Zotero storage location work with ZMI. Perhaps manually doing a search-and-replace in the Zotero export .CSV of the Y:\\........\\.......\\.... path to a local drive path that is itself mapped to your real Y:\\ path? I do not know the correct answer for you. Just note that the two (2) books that were successfully added in [33] did NOT have a file attachment path using: Y:\\, since they were "empty", causing ZMI to add only place-holder .txt documents.

I do not use Symlinks, but in case they are applicable to you, see: How to create Symlinks in Windows 11/10/8/7: https://www.technospot.net/blogs/how...inks-in-vista/


Added: It is possible that you could also be having a security "permissions" error when ZMI in Calibre tries to retrieve the .pdf file from your network (cloud?) storage path: Y:\\........\\.......\\....

Added: Windows uses "\" in paths; Linux and Apple use "/" in paths. "\" is also control character meaning "consider the next character as a normal character, and not as any type of special control character". Perhaps the OS under which your .csv file was created added the backslashes to the path column, which would be wrong since a .csv file is always text. So, does your Y:\\........\\.......\\.... really mean (or should be) Y:\........\.......\....? If so, that is an easy fix. Simply change the .csv data to use \ instead of \\ in that particular column.


D. Regarding the use of random punctuation characters in your Y:\\........\\.......\\.... path that apparently are causing OS file operation errors in ZMI, I will think about what I can do to resolve that issue within ZMI. Note that Calibre itself forbids any such characters in its book paths, just as Calibre does not support networked Calibre Libraries.

E. I need to see a screen-snip of your ZMI Configuration Tab.


Added: Please install the .zip file for ZMI attached below. It has more debug statements, and also will change any ";" into a " " before creating a temp file in ZMI.





DaltonST

[snow-raven]

DaltonST,

I removed ZMI and reinstalled with your zotero_metadata_importer_dev_testing1.zip. I received this error message when trying to import the 31 unimported files: "Error: file atttachments column not found in dict. Cannot proceed."

My debug file & Zotero exports are attached, as well as my Zotfile settings. I have since enabled Zotfile's "Remove Special Characters" setting, but it does not remove semi-colons.

I don't understand your comment that the 2 files that were successfully imported yesterday from the batch of 33 did not have network attachments. The Dowl report WAS attached from: "Y:/HS&E/Permitting/Zotero_References/DOWL HKM/DOWL HKM - 2014 - Preliminary Visual Impact Analysis.pdf" before it was copied to the temp directory during import -"C:\Users\bhughes\AppData\Local\Temp\calibre_v2ypd _qj\n864a5z1_zotero_". It was successfully imported to Calibre. There were also approximately 130 others from the initial batch that also worked, most of which would have had attachments in the same Y:/HS&E/Permitting/Zotero_References directory.

31_ZMIDebug.txt
31_NotSyncedProperly.csv.txt

[snow-raven]

DaltonST,

I tested a fresh set of reports with the new ZMI test version you sent me. All 32 worked perfectly. 2 had no attachments, 30 were stored in Y:/HS&E/Permitting/Zotero_References.

I hope this is helpful.
snow-raven

32New_ZMIDebug.txt
32New_SyncToCalibre.csv.txt

[DaltonST]

You appear to have fixed the Y:\\..\\..\\.. problem, since the .csv file shows only a single \.

The ZMI change to remove any semi-colons solved that path issue.

Sounds like all is well now. If so, I will publish the latest changes to ZMI tomorrow.



DaltonST

[DaltonST]

Version 1.0.77 - 2022-11-30 Miscellany. Minimum Calibre version still 6.0.0.



DaltonST

[stefan5]

DaltonSt, many thanks for this plugin, and for the Extract RIS Citations (my comment is mainly about this last one, since the thread for it is closed on mobileread). My case is as follows: I’m trying to import metadata records from Zotero into Calibre – metadata only, I don’t care about file attachments, since I don’t have any such attachments in Zotero. I’ve tried importing a CSV via ZMI for this task, which works great, except it leaves out significant Zotero fields (or fields I’m interested in), like “Abstract”. I then tried importing a RIS file via Extract RIS Citations, which has the advantage of importing all RIS tags, including the Abstract (AB) one, – BUT, the problem is that Extract RIS Citations only works with RIS files that contain one single record/citation. My RIS files exported from Zotero contain several records/citations per file, and your plugin only extracts the first record in the file. Is it possible for you to code the Extract RIS Citations to handle such multi-record RIS files, in order to extract all records/citations in a RIS file, not just the first one? Thanks.