[GUI Plugin] Zotero Metadata Importer

[DaltonST]

@petzi:

I have attached a proper Zotero CSV file with only column headings.

Your CSV file apparently has different column headings, including perhaps a column with no heading at all. The latter is likely what caused: "Error: file attachments column not found in dict. Cannot proceed."

Please confirm that your CSV file is the generic Zotero export CSV file with a proper first row with the column headings, and that all columns have a heading.

If there are columns to the far right of column "Legislative Body", which is the last column in the generic CSV file, then ensure they have a column heading too.

ZMI reads the first row for the column headings, then maps each CSV column to the correct #zotero_xxxxx custom column in Calibre. It is okay to add new columns the CSV. They just will not be updated in Calibre until they are on the "official list", which is to say, there is a Zotero custom column to map it to. I am going to change ZMI to more gracefully handle corrupt columns. It will still issue the error message, but then will thereafter ignore that corrupt column.

Be sure to always use the most recent version of ZMI. Calibre updates its list of plug-ins to pick from only once an hour at the bottom of the hour plus a few minutes. If you want it sooner, you must go to the Original Post and manually download and install it via Preferences > Plug-ins > Load from file.


Thanks.


DaltonST


n.b. Mobileread.com will not allow you to upload a file with a file extension of .csv, so change it to .txt instead. It will also not allow a .7z extension, so you have to use .zip instead.

[DaltonST]

Version 1.0.9 - 2016-06-27 New CSV-related error messages to assist in correcting CSV data issues.

I hope that you never see these new messages.


DaltonST

[DaltonST]

Version 1.0.10 - 2016-06-27 Major enhancements.

Everyone should upgrade to 1.0.10.


DaltonST

[petzi]

[1] Does anything in the CSV work for you? 0%? 50%? 99%? How many books are in the CSV that you are testing?

No, never. Always 0%. The reported trial with the Windows machine I did over Skype with of friend of mine who has Zotero installed, but not Calibre. I'm out of town and have limited access to PCs and internet.

[2] I have attached the .csv file. I exported it from a collection and left the checkmark "Notes" (or Annotation) unchecked. It is a generic file, I have not touched it, opened it or whatsever. I just zipped it. And I have used the newest ZMI version 1.0.9. (It would be nice if you could include the version number in the ZMI dialog window.)

[4] You might try using Zotfile to rename the books that have the errors to use only basic characters to see if that fixes the problem.

I have done this in some of my test trials. Just to be sure that there is nothing wrong with the file name. I also worked out some trials with just english filenames. (This was the reason why I exported sometimes only two/three Zotero entries – to have complete control.)

In some of my latest trials (starting with version 1.0.6 I believe) I sometimes got different error messages about a "TECHNICAL FAILURE".

calibre, version 2.60.0
ERROR: ZMI: TECHNICAL FAILURE for: calibredb add --library-path=/Users/xxxxx/Documents/Meine E-Books/Calibre/Zotero --identifier zkey:S6IP87AQ "/Users/xxxxx/Library/Application Support/Firefox/Profiles/6kbcf8zv.Standard/zotero/storage/FCQ22QCV/dig_erlass_bl1.pdf"

TECHNICAL FAILURE for: calibredb add --library-path=/Users/xxxxx/Documents/Meine E-Books/Calibre/Zotero --identifier zkey:U48Z5ZX5 "/Users/xxxxx/Library/Application Support/Firefox/Profiles/6kbcf8zv.Standard/zotero/storage/S7JZ3SBF/Internetkompetenz_von SchuelerInnen.pdf"

In these cases I have the relevant files not included in my next trial, e.g. I exported another .csv file without these files. (Attention: these two files are included in the export file I have attached to this post.)

Columns: I have compared your file with my column names. They appear exactly as in my trials. I had no column without a header and I used a generic (default) column setting, when I created the new library. After each trial I deleted the (empty) library and also the folder where the files should go, because there were two metadata files in it. So essentially I started each trial with a complete new session: new library, new .csv-file and so on.

PS.: Due to other obligations I do not know when I will have time to look into this issue again. I hope that I will next Monday online again.

[DaltonST]

@petzi:


Calibredb appears to take in the given filename, and "cleans it" to be harmonious with Calibre. It then adds it to Calibre. However, if several filenames become identical after the cleaning process, then only the very first one will be added to Calibre. Calibre will say "duplicate book", and ignore it. It really is not a duplicate in the Zotero storage folder, but remember that Calibre "cleans it", making duplicates dynamically. Example of leading characters in a filename that calibredb will pretend are not there when given to it via the command-line parameter: "." "_" "._" and similar.


[1] Try renaming your filenames so none start with "." or "_" or any other characters that calibredb will "clean" so that calibredb does not treat the path differently than it was given to it by ZMI per the Zotero CSV. When it treats it differently, it will of course not be able to find the file to import it!


[2] You have many file attachments that have multiple file paths in the same column of the same row, separated by a semicolon (";"). Example:

/Users/peterb/Library/Application Support/Firefox/Profiles/6kbcf8zv.Standard/zotero/storage/EVS5ZD7H/Austria_piaac-erhebung_2011_12.pdf; /Users/peterb/Library/Application Support/Firefox/Profiles/6kbcf8zv.Standard/zotero/storage/PV6N9NSG/idcplg.pdf

This will not work. One (1) .pdf per CSV row per CSV Key (first column).


[3] You have file attachments with single quotes in their filenames. Calibre will "clean" it, and hence never be able to find it. Example:

._[THESIS TITLE GOES HERE] - 'Georgia' Tech Thesis Office.pdf
is treated by calibredb as if it were:
[THESIS TITLE GOES HERE] - Georgia Tech Thesis Office.pdf


[4] Could you please test this very specific file attachment? I can find nothing wrong with it, and it should work perfectly. If it does not, please send me the debug log for that execution of ZMI.

/Users/peterb/Library/Application Support/Firefox/Profiles/6kbcf8zv.Standard/zotero/storage/FG93WTHN/Murray et al_1998_Adult Literacy in OECD Countries.pdf

[5] You have a lot of .html files as attachments. Do you want those imported into Calibre too? Not just .pdf, .epub, etc.?




DaltonST

[petzi]

[4] Could you please test this very specific file attachment?

Didn't work. How can I produce a debug log file? Never done. I invoked "calibre-debug --gui" via the terminal and after that I started the ZMI and produced the error. Then I looked into the terminal window again. But there was no interested information produced:

calibre 2.60 embedded-python: True is64bit: True
Darwin-14.5.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '14.5.0', 'Darwin Kernel Version 14.5.0: Thu Apr 21 20:40:54 PDT 2016; root:xnu-2782.50.3~1/RELEASE_X86_64')
Python 2.7.11
OSX: ('10.10.5', ('', '', ''), 'x86_64')
Successfully initialized third party plugins: DeDRM (6, 4, 3) && Zotero Metadata Importer (1, 0, 10) && Favourites Menu (1, 0, 4) && Goodreads Sync (1, 9, 0) && Find Duplicates (1, 6, 2) && EpubMerge (2, 0, 12) && Annotations (1, 4, 7) && Goodreads (1, 1, 11) && View Manager (1, 3, 2)

What exactly do I have to do for producing a debug log?

[5] You have a lot of .html files as attachments. Do you want those imported into Calibre too? Not just .pdf, .epub, etc.?

No, just files like .pdf, .epub.

[DaltonST]

@petzi:

Calibre is cross-platform. Use Calibre itself to go into debug mode. Do this:

Right-click the Preferences button. Left Click "Restart in Debug Mode".

Calibre will restart, giving you a message. Run ZMI for that specific file. Then, exit from Calibre. Immediately afterwards, a text file with the debug log will pop open. Please zip it and send it.


Thanks.

DaltonST

[DaltonST]

Version 1.0.11 - 2016-06-28 Enhancements.

Please remember to check Calibre status bar in the lower right corner for a message about which of your installed plug-ins have an update available. Click the message to update ZMI.

You should also add the "Plug-in Updater" to your Calibre ToolBar via Preferences.


CALIBRE_TEMP_DIR:

If you implement CALIBRE_TEMP_DIR, then as a minimum make TMPDIR = CALIBRE_TEMP_DIR.

If you want total flexibility, read the Original Post about Special OS Comments, especial item [5].



DaltonST

[DaltonST]

Version 1.0.12 - 2016-06-28 Major Enhancements.

I have changed ZMI so that calibredb never looks into Zotero for the books. Instead, ZMI copies them into a temporary directory, sanitizes their names while it still know what their Zotero Key is, and then points calibredb to that temporary directory using their Zotero Key to create an Identifier with which to match their metadata automatically.

This was necessary due to the complexity of Unicode symbols in some Zotero file attachments (books) that otherwise would never be able to be "found" by calibredb.




DaltonST

[petzi]

Originally Posted by DaltonST

@petzi:

Calibre is cross-platform. Use Calibre itself to go into debug mode. Do this:

Right-click the Preferences button. Left Click "Restart in Debug Mode".

Calibre will restart, giving you a message. Run ZMI for that specific file. Then, exit from Calibre. Immediately afterwards, a text file with the debug log will pop open. Please zip it and send it.


Thanks.

DaltonST

Thank you for the infomation how to switch into debug mode. Attached is the log file.

[DaltonST]

@petzi:

Thank you. Could you now upgrade to Version 1.0.12 of ZMI that was heavily enhanced using your Zotero CSV as test data? Then, do the testing again?


Thanks.



DaltonST

[petzi]

Sorry, again the same error. Log file attached.

[DaltonST]

@petzi,

Thank you. Excellent.

You need to purge all books from your Zotero Calibre Library before running the test book.

Remember, you must populate your Zotero Calibre Library exclusively with ZMI, and ZMI is evolving rapidly. So, you must do what I do. I delete all of my Zotero Calibre Library books and start over with the exported CSV file. Otherwise, you are mixing apples and oranges (different ZMI versions).

Once ZMI reaches steady-state, you will not have to purge the books every time you run a test. ZMI is less than a week old now, and your test data is what is driving its development at the moment. It can handle all of @excaliber's data, but yours...not yet. But soon.


Thanks again.


DaltonST

[petzi]

Actually during the test sessions with you I am always deleting the Zotero Calibre Library and then deleting the folder where still some metadata are stored. I always start form scratch. I do not understand the duplicate message…

Anyway I attached the debug log from the new test iteration.

There is now a more detailed error message. But the hint with the manual update I do not understand. There are no books in my new database to update manually.

[Just another small issue: Whenever I start from scratch I will get the notification from Calibre (calibre-notifier) that some column are missing and I should wait. But actually nothing will happen before I press the ZMI button for “Autovalidate/Generate Current Library…". Only then the missing columns will be generated. No big deal but could user confuse.]

[DaltonST]

@petzi:

Your Calibre library metadata.db appears to need fixing. Calibre > Library Button > Library Maintenance > Check Library. Then, tell it to delete all of the invalid entries it finds.

Do this after you have deleted all of the books.

But actually nothing will happen before I press the ZMI button for “Autovalidate/Generate Current Library…".

It would be dangerous to automatically do it, since the user may not want to automatically do it. So, it waits to continue. As long as the dialog with the "OK" and "Cancel" buttons is shown, ZMI is waiting for you to approve its next action.


DaltonST