![]() |
#1 | |
Custom User Title
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,213
Karma: 63194629
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
|
[FanFicFare] Metadata Management for Fanfiction
As requested by JimmXinu.
Quote:
https://github.com/JimmXinu/FanFicFa...dataManagement Here's one I added (and use sometimes): Code:
Removing redundant character entries Some fandoms have characters of indeterminate gender (e.g. player characters in video games), and authors sometimes add both a general and gender-specific entry. [defaults] exclude_metadata_pre: ## Removes "CharacterName" if Male or Female CharacterName is already present. characters=~^CharacterName&&characters=~^(Male|Female) CharacterName Code:
More selective chapterslashtotal [Archive of Our Own] For Archive of Our Own, this will display '5/10' in its entirety, but only '5' for '5/5' or '5/?'. Replaces the default 'Chapters' on the titlepage. [archiveofourown.org] chapterslashtotal_label: Chapters titlepage_entries: seriesHTML,category,genre,language,characters,ships,status,datePublished,dateUpdated,dateCreated,rating,warnings,chapterslashtotal,numWords,publisher,description add_to_replace_metadata: chapterslashtotal=>^(\d+)/\1$=>\1 chapterslashtotal=>/\?$=> ![]() This could probably also apply to other fandoms, though I couldn't tell you of any off-hand. Code:
## Splitting up Mass Effect into Trilogy and Andromeda category=>^(Mass Effect - Various Authors|Mass Effect \d|Mass Effect - All Media Types)$=>Mass Effect category=>^Mass Effect$=>Mass Effect: Andromeda&&characters=~(Ryder|Jaal Ama Darav) category=>^Mass Effect$=>Mass Effect Trilogy Last edited by ownedbycats; 02-25-2023 at 11:48 AM. |
|
![]() |
![]() |
![]() |
#2 |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,163
Karma: 32196
Join Date: Jan 2007
Location: Anchorage, AK
Device: Sony Reader PRS-505, PRS-650, PRS-T3, Pocketbook HD2
|
Thanks for sharing what you're doing with your metadata.
Right now I'm mostly using replace_metadata to normalize the fandom tags, so I can use generate cover to insert fandom covers. Yesterday, I realized I could do something similar with Genre/freeformtags but it's not without risks and if I went at it full speed ahead my lines of code would reach thousands, which is a bit much. For now, I'm using this to replace genre metadata. Code:
replace_metadata: genre=>.*[Nn]o beta.*=>Not Beta Read genre=>.*[Uu]nbeta.*=>Not Beta Read genre=>.*[Ss]low [Bb]urn.*=>Slow Burn genre=>.*slowburn.*=>Slow Burn genre=>.*[Ff]ix[Ii]t.*=>Fix-It genre=>^[Ff]ix [Ii]t=>Fix-It Code:
exclude_metadata_pre: warnings==Author Chose Not To Use Archive Warnings warnings==No Archive Warnings Apply warnings==Graphic Depictions Of Violence warnings==Creator Chose Not To Use Archive Warnings warnings==Additional Warnings In Author's Note freeformtags,genre==Other Additional Tags to Be Added freeformtags,genre=~^I .* freeformtags,genre=~^[Aa]dditional.* freeformtags,genre=~^[Aa]uthor.* freeformtags,genre=~^[Aa]nd.* freeformtags,genre=~^on .* freeformtags,genre=~^[Aa]lso.* freeformtags,genre=~^[Aa]lmost.* freeformtags,genre=~^NOTE.* freeformtags,genre=~^I'm.* freeformtags,genre=~^this.* freeformtags,genre=~^[Ii]n .* freeformtags,genre=~^[Aa] .* freeformtags,genre=~^[Aa]t .* freeformtags,genre=~^[Ww]ordcount.* freeformtags,genre=~^[Ww]ill .* freeformtags,genre=~^[Ss]ome .* freeformtags,genre=~^[Ss]o .* freeformtags,genre=~^[Oo]h .* freeformtags,genre=~^[Oo]f .* freeformtags,genre=~^[Ww]ell .* freeformtags,genre=~^[Tt]here.* freeformtags,genre=~^[Aa]ka .* freeformtags,genre=~^[Ii]dk .* freeformtags,genre=~^[Aa][Ll][Ll] .* freeformtags,genre=~^[Ii]t's .* freeformtags,genre=~^NOTE.* freeformtags,genre=~^[Mm]ore .* freeformtags,genre=~^[Ll]ike .* freeformtags,genre=~.*[Tt]ags .* freeformtags,genre=~^[Bb]ut .* freeformtags,genre=~.*because.* freeformtags,genre=~^[Hh]e.* freeformtags,genre=~^[Ss]he.* If I thought more author's cared, I'd recommend that they split up their tags to use "Fix-It" "Jedi Order," instead of using "Jedi Order Fix-It." This makes it easier to find all "Fix-It" fics, or all "Jedi Order" you can select on multiple tags in the browser in Calibre. but if they are mushed together it gets lost. Amalthia |
![]() |
![]() |
Advert | |
|
![]() |
#3 | |
Zealot
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 116
Karma: 295672
Join Date: Jul 2021
Device: iPhone
|
Quote:
If I want to look up the freeform tags on Calibre, I can still search for them since they're in a custom column with the long text type. I use MapleRead to read epubs and I want to see the tags by order of their type so I also put a prefix depending on the tag. If it's a category it's [c], for the relationships it's [p], rating [r], status [s], grouped word count [w]. How I add the prefixes: Spoiler:
I remove characters if they're already mentioned in the relationship tag (since you have to manually input each character, I don't cover all of them, only the ones with multiple fics in my library): Code:
exclude_metadata_pre: characters=~CHARACTER_NAME&&ships=~CHARACTER_NAME ## The below line specifically to exclude relationship tags that don't have / or & since some authors put "ship_name relationship" in the relationship tags and I want the tags to be consistent and not redundant ships!~(.*)[/&](.*) Code:
datePublished_format:%%B %%d, %%Y dateUpdated_format:%%B %%d, %%Y add_to_extra_valid_entries:,dateRange include_in_dateRange:datePublished, dateUpdated add_to_replace_metadata: # diff dates 1pmonth 2pday 3pyear 4umonth 5uday 6uyear dateRange_LIST=>([^\s]+)\s+(\d{1,2}),\s+(\d{4}), ([^\s]+)\s+(\d{1,2}),\s+(\d{4})=>\1 \2, \3-\4 \5, \6 # same year 1 2 3 4 5 dateRange_LIST=>([^\s]+)\s+(\d{1,2}),\s+(\d{4})-([^\s]+)\s+(\d{1,2}),\s+\3=>\1 \2-\4 \5, \3 # same month 1 2 3 4 dateRange_LIST=>([^\s]+)\s+(\d{1,2})-\1\s+(\d{1,2}),\s+(\d{4})=>\1 \2-\3, \4 Since I put these all in custom columns, I could also use them as values for the Generate Cover plugin. You could even choose to show only one relationship tag instead of all of them. Here's how that layout was achieved: Code:
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ {#ao3_category} {#ao3_ships:sublist(0,-1,\,)} ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ {#ao3_rating} • {#ao3_words} • {#ao3_chapters} {#ao3_date} |
|
![]() |
![]() |
![]() |
#4 | |
Geek of All Trades
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 55
Karma: 100878
Join Date: Aug 2012
Location: Minneapolis
Device: Kindle Oasis (3rd gen)
|
Love to see what other people are doing. Thanks everybody.
I dislike the FanFicFare's default behavior that dumps a ton of data into the main calibre Tags column/field so I actually edit my personal.ini so FanFicFare doesn't put anything in subject_tags. (I know it works for some people since these fields/columns are easily searched, but I like having things in separate columns/fields.) Instead I have a bunch of custom columns for the data. Relationships column, Fandom column, Warnings column, etc. The most important custom column, that I always create first, is the "Saved Metadata" column so I always have all available data from archives in my library. That way if I mess around with adding new columns or changing them, I can always fill them from Saved Metadata without having to redownload fic. Though it is best to figure out what metadata you value the most and how you want it presented sooner rather than later, especially if you make anthologies (which won't work with Saved Metadata). Or if you want data to look a certain way on title pages. You can inject new generated covers into epubs without too much trouble, but not title pages. Here's my post over in the FanFicFare thread about how I have a tag column that's unedited AO3 freeform tags called Tags (AO3) and another tag column currently named Tags (Edited) which is "only" about 450 tags, some pulled straight from AO3's freeform tags, some modified using replace_metadata. Basically I only keep the tags from AO3's tags that I find useful. Any time I hear someone say they want to try to normalize all the AO3 tags, I point out how futile it is. You can use the data! But my Tags (AO3) column has over 30,000 tags in it (I wish I were exaggerating). Tags (Edited) has 450 or so which is much more manageable to say the least. I keep Tags (AO3) around so I have the data and can look at it or search it if I want to. And to have that information on title pages. (But I mostly ignore it in the tag browser.) Quote:
Because that's a thing I'd like to have on my title pages: the freeformtags as posted. When you put them in a tags column, they're alphabetized which would be fine if people actually used them as tags but of course some authors practically tell stories in the tags (and yes, I like to keep the stories I suppose). |
|
![]() |
![]() |
![]() |
#5 | |
Custom User Title
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,213
Karma: 63194629
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
|
Quote:
Code:
<p><span class='label'>Freeform Tags</span>: <div class='metadata list' id='freeformtags'><ul> <li>Shadowscales</li> <li>Original Character(s)</li> <li>character backstory</li> <li>Implied/Referenced Alcohol Abuse/Alcoholism</li> <li>Alcohol</li> <li>Minor Character Death</li> </ul></div><p> Last edited by ownedbycats; 02-25-2023 at 11:37 AM. |
|
![]() |
![]() |
Advert | |
|
![]() |
#6 | |
Custom User Title
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,213
Karma: 63194629
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
|
Quote:
Hopefully I got it right, I'll ask in the FFF thread. Seems good ![]() Code:
[defaults] custom_columns_settings: editedfreeform=>#yourtagcolumn # 'add_to_' concats on to existing param, thus the leading comma add_to_extra_valid_entries:,editedfreeform include_in_editedfreeform:freeformtags replace_metadata: editedfreeform=>SomeTag=>SomeEditedTag exclude_metadata_pre: editedfreeform=~SomeConversationalTagThatMakesNoSenseWhenSorted Last edited by ownedbycats; 02-25-2023 at 11:58 AM. |
|
![]() |
![]() |
![]() |
#7 | |
Zealot
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 116
Karma: 295672
Join Date: Jul 2021
Device: iPhone
|
Quote:
Mine looks like this: Code:
keep_in_order_category:true keep_in_order_characters:true keep_in_order_additional_characters:true keep_in_order_dateRange:true keep_in_order_fandoms:true keep_in_order_freeformtags:true keep_in_order_genre:true keep_in_order_ships:true keep_in_order_ao3categories:true My include_subject_tags only have the prefixed tags so Calibre's tag browser doesn't get flooded by all sorts of tags that'll just confuse me. Code:
include_subject_tags:prefix_ships,prefix_category,prefix_rating,prefix_status,prefix_wordcount Code:
titlepage_entries:<div class="inline">,seriesHTML,category,rating,warnings,</div>,ships,additional_characters,freeformtags ## metadata ${title}, etc. Make sure to keep at least one space ## at the start of each line and to escape % to %%. titlepage_start:<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>${title} by ${author}</title> <link href="stylesheet.css" type="text/css" charset="UTF-8" rel="stylesheet"/> <meta name="calibre-html-render-data" content='{"size": "A2", "margins": [0, 0, 0, 0]}'/> </head> <body class="fff_titlepage" style="font-size: 2.5vmax;"> <div class="title"> <h1><a href="${storyUrl}">${title}</a> by ${authorHTML}</h1> </div> <hr/> <dl class="tags"> ## ${id}, ${label}, ${value} titlepage_entry: <dt><b>${label}:</b></dt> <dd>${value}</dd> ## ${id}, ${label}, ${value} ## Used with entries listed in wide_titlepage_entries, typically description, storyUrl, authorUrl titlepage_wide_entry: ## When using tables, make these span both columns. wide_titlepage_entries: ## ${id}, ${value} ## For example, if you've set description_label: ## (that is, nothing) titlepage_no_title_entry: ## metadata titlepage_end: <dl class="stats"> <dt><b>Words & Chapters:</b> ${numWords} • ${chapterslashtotal}</dt> <dt><b>Date:</b> ${dateRange}</dt> </dl> </dl> <hr/> <div class="tag-summary-content">${description}</div> </body> </html> Code:
add_to_custom_columns_settings: category=>#ao3_category characters=>#ao3_characters additional_characters=>#ao3_additional_characters dateRange=>#ao3_date dateUpdated=>#ao3_dateupdated freeformtags=>#ao3_additional_tags numWords=>#ao3_words rating=>#ao3_rating ships=>#ao3_ships |
|
![]() |
![]() |
![]() |
#8 |
Custom User Title
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,213
Karma: 63194629
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
|
Unfortunately, keep_in_order only really works for title page; taglike columns always sorted.
|
![]() |
![]() |
![]() |
#9 | |
Zealot
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 116
Karma: 295672
Join Date: Jul 2021
Device: iPhone
|
Quote:
Though I prefer to just use the search bar instead of treating them as a tag so I don't have to clean them up. |
|
![]() |
![]() |
![]() |
#10 |
Custom User Title
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,213
Karma: 63194629
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
|
Ah, that makes sense.
In theory, I think it might be possible to use a gpm template to regex a full-text column and display it as an unsorted column. In practice, I'm not quite good enough at templates or regexes to try this. And it'd be the unmodified tags, which would make it kinda messy. I did use a regex search & replace to populate my #fanficratings column from the longtext rather than run a 'update from column' on everything, though. ![]() search: (?s).*(<p><span class='label'>Rating<\/span>: <div class='metadata' id='rating'>(.*?)<\/div><p>).* replace with: \2 (regex101 for demonstration) Last edited by ownedbycats; 02-25-2023 at 10:50 PM. Reason: adding regex101 link |
![]() |
![]() |
![]() |
#11 | |
Zealot
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 116
Karma: 295672
Join Date: Jul 2021
Device: iPhone
|
Quote:
I wonder if that's faster than filling each custom column via FanFicFare. Last edited by culytera; 02-26-2023 at 08:54 PM. |
|
![]() |
![]() |
![]() |
#12 | |
Custom User Title
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,213
Karma: 63194629
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
|
Quote:
I created the #fanficrating column and set it in FFF, but was too lazy to check 'new only' on all the other columns to avoid replacing everything else. So I manually ran the search & replace to populate it on my existing stories. New/updated stories are automatically filled out. Last edited by ownedbycats; 02-26-2023 at 09:18 PM. |
|
![]() |
![]() |
![]() |
#13 | |
Zealot
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 116
Karma: 295672
Join Date: Jul 2021
Device: iPhone
|
Quote:
![]() Now I'm just trying to optimize the process of downloading and updating the library. Last edited by culytera; 02-26-2023 at 10:40 PM. |
|
![]() |
![]() |
![]() |
#14 |
Custom User Title
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,213
Karma: 63194629
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
|
The other problem with search & replace - longtext column stores original values, so replace_metadatas have no effect. In this specific case, I also wanted to see what all the ratings were before deciding how to 'standardize' them:
Code:
## Ratings rating=>^(K|K\+)$=>General Audiences rating=>^Teen And Up Audiences$=>Teen and Up Audiences rating=>^T$=>Teen and Up Audiences rating=>^Mature$=>Mature Audiences rating=>^M$=>Mature Audiences What I did in this case was use Action Chains to run a single-field edit with this gpm template: Code:
program: ## list_remove_duplicates probably isn't necessary list = list_remove_duplicates($#fanficrating, ','); first_non_empty( if 'Explicit' inlist list then 'Explicit' fi, if 'Mature Audiences' inlist list then "Mature Audiences" fi, if "Teen and Up Audiences" inlist list then "Teen and Up Audiences" fi, if "General Audiences" inlist list then "General Audiences" fi, if "Not Rated" inlist list then "Not Rated" fi ) ![]() Last edited by ownedbycats; 02-26-2023 at 11:58 PM. |
![]() |
![]() |
![]() |
#15 |
Custom User Title
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,213
Karma: 63194629
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
|
For Archive of Our Own: here is my method for copying collections and collection URLs to the series column. I use an Mass Effect fic exchange as an example.
![]() Notes: - This requires a version of FanFicFare released after April 4, 2023 (as of posting date, the latest test version) to make use of a static value in 'include_in.' - If there's both series and collection, you'll need to pick which one you want. Use exclude_metadata_pre to remove the unwanted one using a storyUrl conditional. Code:
[archiveofourown.org] ## This creates an additional 'collectionUrl' entry. add_to_extra_valid_entries:,collectionUrl ## Copies collections to series include_in_series:series,collections ## Creates a placeholder collectionUrl. I chose to use 'abc' for easy replacement (no escapes!). include_in_collectionUrl:"abc" ## Adds collectionURL to series. include_in_seriesUrl:seriesUrl,collectionUrl include_metadata_pre: ## Include specific collections as series. Here, I've included both an exact version (==) and regex (=~). collections==Spectre Requisitions 2023 collections=~(Spectre Requisitions \d{4}|Spec Recs|Spec Recs \d{4}) add_to_replace_metadata: ## I apply transformations to 'collections' but applying them to 'series' should also work collections=>Spec Recs=>Spectre Requisitions ## Here, change the placeholder 'abc' urls to the collection URLs based on a conditional. collectionUrl=>abc=>https://archiveofourown.org/collections/SpecReqs2023&&collections==Spectre Requisitions 2023 exclude_metadata_post: ## If the placeholder isn't changed during replace_metadata, this then removes it to prevent general breakage collectionUrl==abc Last edited by ownedbycats; 04-03-2023 at 10:55 PM. |
![]() |
![]() |
![]() |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Library management issues after editing metadata | dra | Library Management | 26 | 04-11-2020 05:10 PM |
Best strategy for metadata management for Kobo using Calibre? | ceridwen | Kobo Developer's Corner | 16 | 05-13-2019 02:51 AM |
Will automatic metadata management also transfer updated cover images to device? | TheSacredSoul | Library Management | 2 | 01-11-2013 06:05 AM |
Metadata Management on Android not working? | TheStretchedElf | Devices | 0 | 08-08-2012 09:10 AM |
Automatic Metadata Management | gxxshock | Calibre | 2 | 12-28-2008 12:48 PM |