[GUI Plugin] Zotero Metadata Importer

[brokengillou]

You made this fast and it works clean, thanks!
I've done a data copy in LibreOffice Calc from "Link Attachements" to "Extra" in the Zotero exported csv file and imported in Calibre then i let ZMI do the job

Next step i used bulk metadata edit to filter urls in #zotero_extra to #temp with:
Search Field: #zotero_extra
Search for: http://www.sudoc.fr/(.*); http://www.worldcat.org/search\?q=no%(.*)
Replace with: sudoc:\1, worldcat:\2
Destination Field: #temp
options: replace, break

Result Ok
I get the field #temp filled with data like sudoc:147110335, worldcat:3A758470786

Now I would like to append those to the identifiers field with:
Search Field: #temp
Search for: (.*), (.*)
Replace with: \1, \2
Destination Field: identifiers
id type: *
options: append, break, delimiter ", " (wo quotes)

Preview seems Ok
, sudoc:147110335, worldcat:3A758470786
But Apply is not making any change to identifiers...

I must have missed something...

Gilles

[DaltonST]

I think the issue might possibly be that you are trying to create 2 Identifiers simultaneously from a single source field. Identifiers is a complex metadata item, which is why it is not shown in the columns available in Preferences > Add your own columns. It is stored in the database with a key of both book ID and ID TYPE (e.g. isbn or issn or sudoc) and a result field of the value for that combination of book ID and ID TYPE. See the attached image of the metadata.db table "identifiers". That should give you a visual impression of the "destination" that you actually are attempting to populate.

You could create a temporary "Comments Short-Text Like Title" Custom Column each for sudoc and worldcat, then populate each using S&R separately. Then, use S&R for each temporary Custom Column to create a single Identifier for that respective temporary Custom Column. Once both of the Identifiers have been updated, you could delete the contents of the temporary Custom Columns.

Or, you could just keep the temporary Custom Columns permanently, and not make them into Identifiers. Up to you.

Note that if you want to see an Identifier in the Library View, you must create a Composite Custom Column (i.e., built from other columns) with a template such as {identifiers:select(sudoc)} or {identifiers:select(worldcat)} or {identifiers:select(isbn)}.

If you already have a "real" Custom Column for sudoc and worldcat, you would not need a Composite Custom Column to see their values. Your choice.

As usual with Calibre, there are many ways to do something. Your personal preferences should guide you to the correct one for you.



DaltonST

[brokengillou]

I've read some threads quite before so i already managed to add identifiers view from start also custom fields to view ids isolated, no problem thanks.

If amazon and google are stored in identifiers why those sudoc and worldcat should not be stored the same way? Sudoc provides localization and inter library loan, it's a basic need for students.

WorldsWithoutEnd, is it some kind of model extension for automating id type web link (Sudoc is not providing one neither other public libraries i may use)?

Thanks,

Gilles

[aquiaolado]

Hi there!

Is it possible to alter the plugin so it can generate a ZIS Tag L1 so that Zotero recognizes the location of the file through {path}?

Thank you.

[DaltonST]

Sure, but where do you see L1 appear in Zotero? I already have {path} in M2, and it appears in "Extras". I manually added L1 to a test ZMI RIS export file, and after changing the backslashes from / to \ for Win10, Zotero accepts the L1 without any error but does not seem to do anything with it. M2 still appears where it belongs, but nothing else seems to change. What are you seeing if you manually add L1 to your personal ZMI RIS export file?


DaltonST

[aquiaolado]

Originally Posted by DaltonST

Sure, but where do you see L1 appear in Zotero? I already have {path} in M2, and it appears in "Extras". I manually added L1 to a test ZMI RIS export file, and after changing the backslashes from / to \ for Win10, Zotero accepts the L1 without any error but does not seem to do anything with it. M2 still appears where it belongs, but nothing else seems to change. What are you seeing if you manually add L1 to your personal ZMI RIS export file?


DaltonST

Hi there I know nothing about programming, sorry. Anyway I changed the your plugin in 4 rows and reloaded it to Calibre (please see prints attached). Then I created a column in Calibre named #file that has the value for all the books of "file:///". Then I tried to write the Calibre Source in the ZMI: Export RIS so that L1 would return {#file}/{path} or for someone that doesn't know anything about programming would join the "file:///"+"D:/books/etc....".
Problems:
I know nothing about this....
The L1 returns what I wanted but it puts several spaces between the two expressions.
The value of {path} is not the value of the file though, so it doesn't import anything.
I made an experiment exporting a RIS file with only one book from Zotero and then reloading it back to see what was in it, that is how I get to L1 and then to altering your plugin. Anyway it works but I cannot replicate it, and I tried as you can see by now...
What is the value for files in Calibre as the value for path is {path}?
How do I remove the spaces between the two expressions?
Can you still help me? It works with the file exported from Zotero and then reloaded back to Zotero. So Zotero recognizes the instruction and acts accordingly. I even changed the direction of the slashes in the exemple I am sending you...
Can you make sense of all of this? Sorry.

the RIS file that works has this inside:
TY - BOOK
T1 - The Manipulation of Online Self-Presentation
SN - 9781137483416
DO - 10.1057/9781137483416.0001
L1 - file:///D:/Livros do Doutoramento/A. Attrill/The Manipulation of Online Self-Pre (595)/The Manipulation of Online Self - A. Attrill.pdf
ER -

[DaltonST]

Calibre supports 'all' ebook formats such as .epub, etc. The test I did was for an epub, which explains why I saw no results in Zotero. L1 appears to only be used for .pdf files.

When I built ZMI I reverse engineered Zotero's JavaScript import adaptor for generic RIS files with a record type of 'BOOK'. Hence, L1 was not included as a supported tag for Calibre exports since it is not generic. I set M2 as {path}, which works perfectly for real ebooks.

What was the record type for your RIS file exported from Zotero that you immediately reimported? It should appear at the very top of each RIS record.

I might enhance the plugin to export an L1 tag if the Calibre ebook format is pdf if you can answer the questions above and below.

I would still like an answer as to where in Zotero you saw the L1 tag come to rest. What does it do for you? Exactly?



DaltonST

[aquiaolado]

Hi there,

Did you see the picture i sent? It has a view of each thing I did.

First question's answer:
The RIS record type is the one present in the record I reproduced at the end of the reply (the one that works) and is "TY - BOOK".

Second question's answer:
The L1 tag adds the pdf file to the record, as you can see in the picture attached.

Finally, and maybe I should have stated this in the beginning, what I am trying to achieve:

• I am a PhD student.
• I "grabbed" near 4000 books from the internet.
• I "passed" them through Calibre so I can have a good deal of metadata e the various fields, with correct ISBN, etc.
• I would like to export them changing the name and the metadata, and ending with only one pdf file, instead of the three files (file+cover+metadata), so that I can import them to a reference manager (Endnote, Mendeley, Zotero) - !!!this is not working!!! So, instead, I am trying to export them from Calibre to Zotero with the pdf file attached. The thing is that your plugin works perfectly to export all the metadata but it misses the L1 (as you already justified why). So I was trying to solve it on my own, but I know nothing about programming.
• Now I think that I can, after I add the files to Zotero, export them to Mendeley, add the feature that they have of identifying The TOC (Table of Contents) and in the end have a set of files with the correct ISBN, the Name of the File that I want, the several metadata and tags and TOC and start from there to analyse my bibliography and querying it, etc. Ultimately I want to add them to a content analysis software like MaxQDA or NVivo and start working on my thesis with a solid database.
• Sorry for this.

[DaltonST]

Looks good. I will add L1 to ZMI's RIS export function for Calibre books that have a .pdf format. I should upload the new ZMI version later today.



DaltonST

[DaltonST]

aquiaolado:

Although I have attached is an updated version of ZMI supporting L1 as an RIS export tag, I cannot reproduce your results.

I very strongly suspect that it is due to the fact that you have previously manually linked or attached the Calibre ebook file to your Zotero Item, so when Zotero exports its own RIS file, it "knows" about it already. When you immediately re-import what it just exported, it can do so cleanly for that reason.

A revised ZMI RIS export file for a .pdf is shown below:

Spoiler:

TY - BOOK        
AU - Ackerman, Diane
A2 -         
A3 -         
A4 -         
AB -         
AV -         
CN -         
CR -         
CY -         
DA - 2011-11-05        
DB - Calibre        
DP -         
ET -         
KW -         
LA - por        
L1 - file:\\\S:\Calibre\CalibreZotero\Ackerman, Diane\The Zookeeper’s Wife (5557).pdf
M1 -         
M2 - S:/Calibre/CalibreZotero/Ackerman, Diane/The Zookeeper’s Wife (5557).pdf        
N1 -         
N2 -         
NV -         
PB - Norton        
RN -         
RN -         
RN -         
RN -         
RN -         
RN -         
RN -         
RN -         
RN -         
RN -         
SN - 9780393069358        
SP -         
ST -         
T1 - The Zookeeper’s Wife        
T2 -         
UR -         
VL -         
Y2 -         
ER -

If you wish to manually change anything within the attached .zip plugin file, search for "D:/Livros do Doutoramento/A. Attrill/The Manipulation of Online Self-Pre (595)/The Manipulation of Online Self - A. Attrill.pdf" in the same .py file that you were previously playing with, zmi_dialog.py. However, removing or adding "file:///" or "file:\\\" makes no difference in my testing, because I am starting with a Zotero lacking any of my test data, whereas you are exporting your test data and then immediately re-importing it.

In short, you want to use the ZMI RIS file to automatically link/attach your Calibre .pdf files within Zotero. That would be wonderful if it worked, but as far as my testing can determine, the Zotero input adapter for RIS files does not support that unless it is doing what you did (export an existing attachment to an RIS file and then immediately re-importing it).

This version is just for you to manually download. I have not officially published it to Calibre yet.

Added: you likely corrupted your "Zotero Metadata Importer.json" file for its value for the key of "L1". You should open that file with a good UTF-8 enabled text editor, delete the entire L1 line, save, and exit. When you next run ZMI, it will default L1 to the proper value, "{path}", automatically. If you read the ToolTips for the ZMI RIS Export Tab, you will understand why you cannot add random colons in those values.



DaltonST

[aquiaolado]

Hi there DaltonST,

Besides the fact that I know little about this, I think i did not explain myself well, sorry.

When we are in the "ZMI: Export RIS" in the "L1" row how can i put an expression that goes like this:

"file:\\\"+"{path}"+"{filename}"

whereas

"file:\\\"
I turned it around and created a column in Calibre with the content "file:\\\" for all entries and it is called "file" and then I put {#file} in the "ZMI: Export RIS" and it returns in the RIS file after "L1 - " the expression "file:\\\"

"{path}"
Returns, as normal, the path of the file connected to each of the rows - but it doesn't return the filename. It just returns the path, without the filename

"{filename}"
I don't know how to ask Calibre for the filename. As well as path there must be an expression that returns the filename in Calibre.

So the problem is not in Zotero, maybe you are misreading it. The problem is that I cannot build a full expression that redirects Zotero to the actual file

I can build file:\\\ + {path} but it puts some spaces between the two expressions. And more than that I don't know how to complete it with the actual filename.

Sorry. Hope you can understand me.

PS: Is there a way for us to do a conference online?

[DaltonST]

As I mentioned in my last post, you must read the ToolTips. It explains the syntax for the value fields. To combine 2 templates, you must separate them with a slash. Example: {#mysomething}/{#myotherthing}

Regardless, Zotero will not be redirected to the actual file because Zotero did not export the .ris file; it was created within Calibre.

Please read my prior post. What you are trying to do appears to be a waste of time for the reasons I stated.

I have tweaked the .zip file attached to the previous post. You should reinstall ZMI from that .zip file.



DaltonST

[aquiaolado]

I know what you are saying. I did that. I read all the instructions. I join {#file}/{path}. It returns what I want but puts spaces between the expressions. But beside that I still need {#file}/{path}/{something that would return the filename from Calibre}.
Can we speak directly?

The Zotero works fine with the file you plugin/Calibre produces and with a file that was never inside zotero.

If I produce the RIS file with your plugin and then alter it manually to a file that zotero does not know it works.

[DaltonST]

You are not listening. What you are doing is a waste of time. Read my prior post.

Added: you should manually edit your exported-from-Calibre .ris file to make it how you like, and then import it into Zotero. You will see that nothing good happens for the reasons I stated.

[aquiaolado]

Sorry. I know you think I am not making sense but please give me some time and let me work with you. It does work, i just don't know how to do it, but you know how to. Can we talk directly?