[GUI-Plugin] Library Codes-SRU

[davidjoseph1]

[GUI Plugin] Library Codes-SRU

A forked version of the original Library Codes by DaltonST: see posted updates on the legacy plugin thread.

Functions and Features:

[1] Automatically creates new Custom Columns (if desired) for Dewey Decimal Code and/or Library of Congress Code and/or OCLC-OWI and/or FAST Tags.

[2] A. Derives the desired Library Codes using a book's ISBN or ISSN [Selected Books].
B. Derives the desired Library Codes using a book's Author and Title [Selected Books].

[3] NOT WORKING YET Optionally creates Identifiers: OCLC-OWI; OCLC-WORLDCAT; and VIAF_AUTHOR_ID. This allows the use of composite Custom Columns to be shown in the Book Details as a hot-link to the respective URLs.

[4] Optionally Converts ISBN-10 to ISBN-13 [Selected Books]. This function will also automatically "sanitize" ISBNs to remove any dashes, etc. even if the ISBN is already an ISBN-13.

[5] Optionally extracts ISSNs from PDFs for Magazines and Periodicals. The other LC functionality then may be executed for the PDFs for which ISSNs were extracted. View the LC Menu ToolTips for more information.

[6] Optionally maps either a LCC or a DDC for each book to a 'Genre', and updates a Text custom column with that mapping. LC comes preloaded for DDC mappings, but a template .csv file for LCC mappings is attached below.

Please read the ToolTips in the LC customization settings.

For a full Open Source DDC listing for your own use: https://raw.githubusercontent.com/Jo...master/fdc.txt

Code décimal de Dewey en français: https://www.mobileread.com/forums/sh...4&postcount=94

Version History[/B][/U]

Spoiler:

Version 2.0.4 - 2024-08-01 Author-Title search expression tweaked for greater accuracy.
Version 2.0.3 - 2024-07-31 Added Author-Title search functionality
Version 2.0.2 - 2024-07-22 Added 1.6 second pause to avoid server congestion timeout and renamed plugin inside init.py
Version 2.0.1 - 2024-07-11 Redesigned Classify scrape to search for matching records at the Library of Congress SRU server and parse MODS3.8 to return LCC and DDC. still working on FST, VIAF, and OCLC
Version 1.0.65 - 2023-04-16 Bug fix. Change in multiple-response re-do trigger.
Version 1.0.64 - 2023-04-15 Automatically get oclc-wi if needed to obviate need to get oclc-wi via Author/Title. This automatically turns on saving all LC Identifiers, regardless of customization.
Version 1.0.63 - 2023-04-14 Miscellany
Version 1.0.62 - 2023-04-13 Miscellany
Version 1.0.61 - 2023-04-12 Add back self.detect_isbn_changes_to_refresh.
Version 1.0.60 - 2023-04-06 LCEAD preferences and widgits removed from within Customization.
Version 1.0.59 - 2023-04-06 Miscellaneous technical improvements. Also, the LCCN Identifier values are automatically converted so that they may be used as-is with the URL to the Library of Congress Permalinks to the book's author. The format for one possible Permalink is: https://lccn.loc.gov/n80126289 where the n80126289 is the LCCN value from Library Codes. A better link to it is: https://id.loc.gov/authorities/names/n80126289.html You may set either of these links in Calibre > Preferences > Look & Feel > Book Details.
Version 1.0.58 - 2023-04-05 Enhancments to increase speed of execution of LC, plus other technical improvements.
Version 1.0.57 - 2023-04-05 Enhancement to add a new menu item to use a book's Author & Title to download its OCLC-WI Identifier, which in turn can be used in lieu of a missing ISBN/ISSN to download the same metadata (DDC, LCC, FAST Tags, etc.) that ISBN/ISSN would make available if they were not missing. Enhancements to download more metadata. Changes to ToolTips. Changes to Customization.
Version 1.0.56 - 2023-04-01 Streamlined the new "no Classify API", webscraping-based version of LC, to remove deprecated functions requiring the old Worldcat API. New: added the creation of 3 Identifiers: OCLC-OWI; OCLC-WORLDCAT; VIAF_AUTHOR_ID. LC Customization revised to be consistent with both of the previous.
Version 1.0.55 - 2023-03-31 No API: Life-Support Replacement for API that requires a Key since December 2022.
Version 1.0.54 - 2023-01-18 Qt.core.
Version 1.0.53 - 2022-07-14 Enhancements to the "eBook"/"Book" criteria for multi-work responses.
Version 1.0.52 - 2022-04-19 If a return code of '4' is received, and its attributes include 'wi' (not 'owi'), create a new RC of '2' using the syntax http://classify.oclc.org/classify2/Classify?wi=nnnnnnn
Version 1.0.51 - 2022-03-19 Qt6 compatibility. Minimum Calibre version 6.0.0
"eBook" added to "Book" as a possible work type for multi-work responses from Classify's API.
Version 1.0.50 - 2020-12-22 Miscellaneous Technical changes.
Version 1.0.49 - 2020-12-15 Miscellaneous technical improvements.
Version 1.0.48 - 2020-11-24 Favourites plugin accommodation.
Version 1.0.47 - 2020-08-11 Qt tweaks for small displays.
Version 1.0.46 - 2020-07-22 New menu option to remove redundant LCEAD Tags that already exist as FAST Tags for the same selected books.
Version 1.0.45 - 2020-07-17 Major webscraping rewrites due to Worldcat site changes. Configuration Dialog greatly enhanced for both validation of user settings and ToolTips.
Minimum Calibre Version remains 4.6. BeautifulSoup 3 is no longer supported. Tested in Calibre 5 Beta 4.99.7 and required no changes.
Version 1.0.44 - 2020-07-12 Miscellaneous technical changes to customization and ToolTips.
Version 1.0.43 - 2020-07-02 Additional DEBUG statements added; Technical change to customization.
Version 1.0.42 - 2019-12-30 Technical changes after Python 3.8 testing with Calibre 4.99.2
Version 1.0.41 - 2019-04-30 Python 3 compatibility
Version 1.0.40 - 2019-04-27 Python 3 compatibility & Calibre Version 3.41.3 pdf-to-html compatibility (ISSN extraction from PDFs)
Version 1.0.39 - 2019-04-13 Use standard Calibre's BeautifulSoup, and choose bs3 .findAll or bs4 .find_all dynamically based on its attributes
Version 1.0.38 - 2018-10-15 Version 1.037 functionality has been removed in favor of using the Job Spy plugin's Scrub Tags tool.
Version 1.0.37 - 2018-10-14 LCEAD Tags and FAST Tags: option to prepend any specified prefix to each Tag.
Version 1.0.36 - 2018-10-12 FAST Tags: more work.
Version 1.0.35 - 2018-10-11 FAST Tags added to LC Customization. FAST retrieved as Tag-Like.
Version 1.0.34 - 2018-05-17 Function "Find Alternatives to a Non-Responsive ISBN" has been redesigned due to current Google Search Books API restrictions.
Version 1.0.33 - 2018-05-16 Function "Extraction of ISSN from PDF periodicals" changed to reflect Calibre changes to its pdf-to-html function.
Version 1.0.32 - 2018-05-15 Techncial and functional enhancements to ISBN scrubbing.
Version 1.0.31 - 2018-05-13 Enhanced scrubbing of ISBNs.
Version 1.0.30 - 2017-05-21 Technical changes.
Version 1.0.29 - 2017-05-11 Technical changes for Calibre 3.0.0 compatibility.
Version 1.0.28 - 2017-05-05 Allow themes with user-defined icons.
Version 1.0.27 - 2016-06-24 Compatibility improvement for use with Linux and OSX
Version 1.0.26 - 2016-05-26 Bug fix.
Version 1.0.25 - 2016-05-25 Technical changes.
Version 1.0.24 - 2016-04-02 DDC Pie Charts
Version 1.0.23 - 2016-03-26 Attempt to solve OSX subprocess issue.
Version 1.0.22 - 2016-03-23 Technical changes.
Version 1.0.21 - 2016-03-22 Revised LCC to Genre mapping logic, plus a corresponding template CSV file.
Version 1.0.20 - 2016-03-21 New: Updating of #genre (or any other text custom column) by mapping the DDC or LCC to Table _lc_genre_mapping, which is automatically seeded with over 2,470 DDC rows.
Version 1.0.19 - 2016-02-14 Miscellaneous enhancements.
Version 1.0.18 - 2016-02-13 Technical changes to improve performance in extracting ISSNs from PDFs.
Version 1.0.17 - 2016-02-13 New: ISSN Extraction from Magazine/Periodical PDFs for use in deriving DDC, LOC, etc.
Version 1.0.16 - 2016-01-30 Technical changes to improve performance in webscraping.
Version 1.0.15 - 2015-12-04 New Menu Option: Attempt to Substitute Non-Responsive ISBN Using Author/Title [Selected Single Book]; Technical tweaks.
Version 1.0.14 - 2015-12-02 Many enhancements, especially to using 'OCLC-OWI" to retrieve 'OCLC' which in turn is used to retrieve 'LOC_LCCN'. Menu and ToolTips changes. BISAC-type LCEAD column data for Fiction is now retrieved (Non-Fiction already was).
Version 1.0.13 - 2015-11-30 New Identifier: 'LOC_LCCN', which is the 'Library of Congress-LCCN'. For books assigned LOC LCCNs after the year 1999, they (probably) have a permalink of: http://lccn.loc.gov/NNNNNNNNNN where the LCCN is a 10-digit number starting with the year (2NNN)
Version 1.0.12 - 2015-11-25 Technical tweak.
Version 1.0.11 - 2015-11-25 Additional LCEAD using OCLC. Most useful for Factual/Non-Fiction. Since LCEAD column is Tag-Like, any 'commas' have been replaced with the 'comma-like' Unicode symbol 'ˏ' that does not confuse Calibre.
Version 1.0.10 - 2015-11-24 New Identifier: OCLC (Not the same as OCLC-OWI). OCLC is extremely useful in retrieving LCEAD data.
Version 1.0.9 - 2015-11-23 New Optional Custom Column: LC Extra Author Details (LCEAD), which is a Tag-Like column that is updated with additional author-related information. It is automatically updated when deriving DDC/LCC, but can be separately updated via its own menu option that uses OCLC-OWI to obtain the LCEAD values.
Version 1.0.8 - 2015-11-21 Deriving ISNI and LCCN Identifiers from VIAF Author ID: speed of download improved significantly.
Version 1.0.7 - 2015-11-20 New option to delete all non-Library Codes Identifiers.
Version 1.0.6 - 2015-11-20 2 New Identifiers: ISNI and LCCN derived from VIAF Author ID to enable hot-linking to ISNI.org and Worldcat.org, respectively. This is done whenever a OCLC-OWI Identifier is otherwise created.
Version 1.0.5 - 2015-11-19 Creation of 2 Identifiers, VIAF.org Author ID and Library of Congress Author Name ID, whenever OCLC-OWI Identifier is otherwise created. This allows the use of composite custom columns to hot-link to the respective URLs to view extended Author and Book information.
Version 1.0.4 - 2015-11-17 New option to convert ISBN-10s to ISBN-13s.
Version 1.0.3 - 2015-11-16 New option to derive DDC and/or LCC indirectly using Author/Title via OCLC-OWI Identifiers for cases where ISBN is not useful or reliable.
Version 1.0.2 - 2015-11-13 Menu changes.
Version 1.0.1 - 2015-11-12 Technical tweak.
Version 1.0.0 - 2015-11-03 Initial release.

[davidjoseph1]

I want to make a couple of use notes:

1) there is a hard-coded per IP query limit of forty requests per minute. Exceed that at the lx2.loc.gov:210/LCDB SRU server, and it will lock you out and not respond to any requests at all for fifteen minutes or so. The response time is so quick that you can exceed that rate. When I modified SRU, I put in a five second pause so that the rate limit would not be exceeded.

2) The record matching is brittle against identifier. The value-add of the OCLC Classify backend was the wideness of applicable responses, and the Library of Congress look up will fail against Most e-book isbns. My suggestion, until I modify library codes further to accept title author ISBN 3-tuple searching, Is to use SRU to get an applicable ISBN, which can be served with fully adequate Dewey and LC call number.


When I figure out how to get title author going o I will update.


Future efforts will also include getting the field 650 and field 655 subject headings, implementing an authority look up for author names against the name and subject authority servers at the Library of Congress, getting VIAF Authority records returned, and figuring out someway to Horse, the free OCLC worldcat web lookup to get OCLC worldcat numbers. However, without any subscription to OCLC Metadata, Which is not possible for non-library entity, it is very hard to do anything with that world, cat number, except for a slightly faster search on the free OCLC world cat page.

[BetterRed]

This will not appear in calibre->Preferences->Plugins until an entry for it is inserted into the Plugin Index sticky thread at the top of this forum.

Suggest @pdurrant should do that.

BR

[JSWolf]

Why fork the plugin when changes like this should be in the original thread?

[davidjoseph1]

Originally Posted by JSWolf

Why fork the plugin when changes like this should be in the original thread?

Changes like this are in the original thread, and after I posted the rewrite it was suggested that I post a new thread because DaltonST has been awol for more than a year, and there was no way to add it to the in-application distribution channel without a new plugin thread.

[stan999]

dude, you are a God send!

[Nathan Brazil]

I see your fork and the original as updates. The original is in deprecated status. Should it be removed?

[davidjoseph1]

Originally Posted by Nathan Brazil

I see your fork and the original as updates. The original is in deprecated status. Should it be removed?

It's named the same in the init.py, so just leave it until I name it differently in an update. The version number ought to be the same in both.

I ought to update it soon to put in a two-second pause so it doesn't overwhelm the query limit on the LOC server.

[davidjoseph1]

Originally Posted by Nathan Brazil

I see your fork and the original as updates. The original is in deprecated status. Should it be removed?

Name is changed, 1.6 second pause added, so I think? you can remove the old version without removing the new one.

[foosion]

Would it be possible to do a lookup based on title and author, maybe with a fuzzy match (there are some fuzzy match libraries on pypi)? I'm just looking to translate to a top-level genre, if that matters.

I see there's a plugin to extract ISBNs: https://www.mobileread.com/forums/sh...d.php?t=126727

[davidjoseph1]

Originally Posted by foosion

Would it be possible to do a lookup based on title and author, maybe with a fuzzy match (there are some fuzzy match libraries on pypi)? I'm just looking to translate to a top-level genre, if that matters.

I see there's a plugin to extract ISBNs: https://www.mobileread.com/forums/sh...d.php?t=126727

I'm working on that right now. That was the value-add of Classify - the relation of a number of different isbns against one another. My workaround right now is to use a metadata plugin called SRU which does do that author title lookup at the LOC and provides a good ISBN with LCCN, which gets the lcc call number information. Works about eighty-five percent of the time.

this is the current sketch of code I have to apply author title search. Not working yet.

[darthyoda6]

Originally Posted by foosion

I see there's a plugin to extract ISBNs: https://www.mobileread.com/forums/sh...d.php?t=126727

That extracts the isbn from the book file, if it can find it in either the top pages or the bottom pages of the book.

[davidjoseph1]

New Version: author-title search works.

[teditor]

FAST tags aren't updating for me. Possible I have a setting wrong? Or is it not working right now? The rest of it is working great, and I'm so grateful to you for resurrecting this plugin.

[davidjoseph1]

Originally Posted by teditor

FAST tags aren't updating for me. Possible I have a setting wrong? Or is it not working right now? The rest of it is working great, and I'm so grateful to you for resurrecting this plugin.

Thank you. I've been frustrated by lack of the tool too.

FAST tags (which are an OCLC thing, FAceted Application of Subject Terminology) will not be working. Instead, I will use LCSH subject headings for the same thing. But it's not working yet.