[GUI Plugin] FanFicFare

[JimmXinu]

If FFF is looking in the wrong cache directory, search your personal.ini, maybe you have more than one browser_cache_path setting.

If you have open_pages_in_browser:true, and the pages FFF wants cached are opening in the default browser, that's how it works.

You could try changing Calibre's openers_by_scheme tweak, but the last report I had from some one trying that, it didn't work.

[fishing_forever]

I've been trying to use the workaround and set the browser_cache_path in defaults. It opens the fic link in my default browser (Firefox), but then I get this error:

HTTP Error in FFF 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)'(428)

I tried setting the browser_cache_age_limit to -1 but that didn't do anything.

When I try to update the same fic again with FFF, it doesn't open it in my browser a second time.

[Chrystalline]

Ah, I tried adding open pages in browser, but when I took it back out, it was stuck. Closing and restarting got the FFN working again. I've never gotten it working with AO3.

Does it make a difference which URL is used? I've tried:
https://archiveofourown.org/works/57...full_work=true which is how it appears when I open the page in the browser.
https://archiveofourown.org/works/57094942 is how it links in my notifications.
https://archiveofourown.org/works/57...56639#workskin is the third chapter if I change it to view chapter by chapter in the browser.

I admit I've copy/pasted a lot of ini code from other people, because I don't fully understand what does what, so I don't know if something in here is causing the problem.

Spoiler:

[defaults]
## [defaults] section applies to all formats and sites but may be
## overridden at several levels.  Example:

## [defaults]
## titlepage_entries: category,genre, status
## [www.whofic.com]
## # overrides defaults.
## titlepage_entries: category,genre, status,dateUpdated,rating
## [epub]
## # overrides defaults & site section
## titlepage_entries: category,genre, status,datePublished,dateUpdated,dateCreated
## [www.whofic.com:epub]
## # overrides defaults, site section & format section
## titlepage_entries: category,genre, status,datePublished
## [overrides]
## # overrides all other sections
## titlepage_entries: category
keep_in_order_characters:true
keep_in_order_category:true
keep_in_order_freeformtags:true
keep_in_order_genre:true
keep_in_order_fandoms:true
keep_in_order_ships:true
characters_label:Characters
ships_label:Relationships
series_label:Series
seriesUrl_label:Series URL
extratags_label:Extra Tags
replace_hr: false
include_images:true
no_image_processing: true
is_adult:true


## As a work around for certain sites blocking automated downloads,
## FFF now offers the ability to look for pages in your Chrome(or
## Chromium-derived) or Firefox browser's cache.  Requires both
## use_browser_cache:true and browser_cache_path to be set.
##
## browser_cache_path needs to be set to the location of YOUR browser
## cache.  Here are a few examples for different OS.  Note that all
## have YOUR user name in them somewhere as well as a Profile name,
## frequently Default.  Make sure you have personalized (and
## uncommented) ONE browser_cache_path setting.

## Note also that browser_cache_path goes under [defaults] because
## browser cache is read once and shared between all downloads for
## performance.

## Further note that there is a time cost to first reading the browser
## cache on each download session (not Calibre session)
## windows:
### Chrome:
#browser_cache_path:C:\Users\YourUser\AppData\Local\Google\Chrome\User Data\Default\Cache
#browser_cache_path:C:\Users\YourUser\AppData\Local\Google\Chrome\User Data\Profile 1\Cache
### Firefox
#browser_cache_path:C:\Users\YourUser\AppData\Local\Mozilla\Firefox\Profiles\ZjwI7Fo4.default\cache2

### FFDL
# browser_cache_path:C:\Users\YourUser\AppData\Local\fanficdownloader\cache\QtWebEngine\Default\Cache

### Opera:
browser_cache_path:C:\Users\YourUser\Desktop\Opera\Data\profile\data\Default\Cache\Cache_Data

## It's common for browser cached files to be kept for several hours
## or more.  You can limit the age of cached files FFF will use from
## browser_cache_path with browser_cache_age_limit.  Only cached files
## that were downloaded within 'browser_cache_age_limit' hours will be
## used.  If set to -1, all cached files will be used.
browser_cache_age_limit:-1

[archiveofourown.org]

## In order to get bookmarktags and bookmarksummary, you need to login
## all the time. This defaults to off to save time and network
## traffic. Requires valid AO3 username and password when true.
always_login:false

use_browser_cache:true
use_browser_cache_only:true

## AO3 offers a page to download the entire work at once. This can
## save a lot of network traffic if there are many chapters.
## True by built-in default.
#use_view_full_work:true

## AO3 allows users to archive stories they didn't write in certain
## cases.  These are indicated by showing a byline such as:
## 'Orig Author [archived by Archivist Author]'
## If use_archived_author is set true (and there's only one author
## listed), author will be set to 'Orig Author' instead of 'Archivist
## Author'.  authorUrl will still point to the Archivist Author's page.
use_archived_author:true

## Some adapters collect additional meta information beyond the
## standard ones. They need to be defined in extra_valid_entries to
## tell the rest of the FanFicFare system about them. They can be
## used in include_subject_tags, titlepage_entries,
## extra_titlepage_entries, logpage_entries, extra_logpage_entries,
## and include_in_* config items. You can also add additional entries
## here to build up composite metadata entries.
## archiveofourown.org, for example, fills genre (a standard
## entry) as the composite offreeformtags, ao3categories in
## include_in_genre. If there's ever more than 4 series, add
## series04,series04Url etc.
extra_valid_entries:fandoms, freeformtags, freefromtags, ao3categories, comments, chapterslashtotal, chapterstotal, kudos, hits, bookmarks, collections, byline, bookmarked, bookmarktags, bookmarksummary, bookmarkprivate, bookmarkrec, restricted, series00, series01, series02, series03, series00Url, series01Url, series02Url, series03Url, series00HTML, series01HTML, series02HTML, series03HTML
fandoms_label:Fandoms
freeformtags_label:Freeform Tags
freefromtags_label:Freeform Tags
ao3categories_label:AO3 Categories
comments_label:Comments
chapterslashtotal_label:Chapters/Total Chapters
chapterstotal_label:Total Chapters
kudos_label:Kudos
hits_label:Hits
collections_label:Collections
## Count of bookmarks on story by all users
bookmarks_label:Bookmarks
## Tags & Summary from *your* bookmark on the story. Only collected
## if always_login:true
#bookmarked_label:I Bookmarked Story
#bookmarktags_label:My Bookmark Tags
#bookmarksummary_label:My Bookmark Summary
#bookmarkprivate_label:My Bookmark Private
#bookmarkrec_label:My Bookmark Rec
#restricted_label:Restricted to Registered Users
#series00HTML_label:Series
#series01HTML_label:Series2
#series02HTML_label:Series3
#series03HTML_label:Series4

## Assume entryUrl, apply to "<a class='%slink' href='%s'>%s</a>" to
## make entryHTML.
make_linkhtml_entries:series00,series01,series02,series03

## AO3 doesn't have anything it calls 'genre'.  The adapter used to be
## hardcoded to include the site specific metadata freeformtags &
## ao3categories in the standard metadata field genre.  By making it
## configurable, users can change it.
include_in_genre: freeformtags, ao3categories

## AO3 uses the word 'category' differently than most sites.  The
## adapter used to be hardcoded to include the site specific metadata
## fandom in the standard metadata field category.  By making it
## configurable, users can change it.
include_in_category:fandoms

## freeformtags was previously typo'ed as freefromtags.  This way,
## freefromtags will still work for people who've used it.
include_in_freefromtags:freeformtags

## chapterslashtotal contains 1/3 or 1/1 or 3/? etc as reported by AO3.
## chapterstotal is just the total chapters part after the /
include_in_chapterstotal:chapterslashtotal.NOREPL
add_to_replace_metadata:
chapterstotal=>^([0-9])+/(.+)$=>\2

## adds to titlepage_entries instead of replacing it.
extra_titlepage_entries: fandoms,freeformtags,ao3categories,comments,chapterslashtotal,chapterstotal,kudos,hits,bookmarks,bookmarked,bookmarktags,bookmarksummary,series01HTML,series02HTML,series03HTML,byline

## adds to include_subject_tags instead of replacing it.
extra_subject_tags:fandoms,freeformtags,ao3categories

## extra_valid_entries defines valid custom metadata entries.
## Need to use add_to_ because defaults.ini already defines several
## under [archiveofourown.org].  Yes, the leading , is needed
## See https://github.com/JimmXinu/FanFicFare/wiki/INI-File#add_to_keyword
add_to_extra_valid_entries:,platships,romships

## Labels used if included in titlepage_entries or logpage_entries
platships_label:Platonic Relationships
romships_labels:Romantic Relationships

## essentially copy ships to platships and romships.  
## **After** replace_metadata and in/exclude on ships so
## and edits to ships will be applied to both still.
## Use ships.NOREPL to not get ships edits.
include_in_platships:ships
include_in_romships:ships

## Include  **only** platships containing & and romships containing /
## & html entity is used for & in FFF entries for historical reasons.
include_metadata_pre:
 platships=~&
 romships=~/

## not necessary, but can be useful to see this part is working if nothing else
add_to_titlepage_entries:,platships,romships

## put platships & romships into their custom columns.
## Quite a few ini parameters are documented primarily in
## the defaults.ini file
## See https://github.com/JimmXinu/FanFicFare/blob/main/calibre-plugin/plugin-defaults.ini#L345
add_to_custom_columns_settings:
platships=>#platships
romships=>#romships
kudos=>kudos
hits=>hits
bookmarks=>bookmarks
comments=>comments

## AO3 chapters can include several different types of notes.  We've
## traditionally included them all in the chapter text, but this allows
## you to customize which you include.  Copy this parameter to your
## personal.ini and list the ones you don't want.
#exclude_notes:authorheadnotes,chaptersummary,chapterheadnotes,chapterfootnotes,authorfootnotes

## AO3 is blocking people more aggressively.  If you download fewer
## stories less often you can likely get by with reducing this sleep.
slow_down_sleep_time:20

## AO3 allows users to archive stories they didn't write in certain
## cases. These are indicated by showing a byline such as:
## 'Orig Author [archived by Archivist Author]'
## If use_archived_author is set true (and there's only one author
## listed), author will be set to 'Orig Author' instead of 'Archivist
## Author'. authorUrl will still point to the Archivist Author's page.
#use_archived_author:false

## AO3 chapters have 'date' available for each chapter.
## These can be used with custom output (see
## https://github.com/JimmXinu/FanFicFa...i/CustomOutput ) or with
## chapter_title_*_pattern settings.
## Examples for html & epub:
#[html]
#tocpage_entry:
# <a href="#section${index04}">${chapter}</a> ${date}<br />
#[epub]
#tocpage_entry:
# <a href="file${index04}.xhtml">${chapter}</a> ${date}<br /><br />

## The 'date' value for chapters mentioned above can be formated with
## datechapter_format. Otherwise it will default to
## datePublished_format
#datechapter_format:%%Y-%%m-%%d

## (Plugin Only) - You can also populate calibre custom columns with
## the site specific metadata using custom_columns_settings (but only
## if 'Allow custom_columns_settings from personal.ini' is checked in
## the plugin GUI config.) There are three parts, the entry name,
## then the label of the calibre custom column, then (optionally) a
## 'mode'. 'r' to Replace any existing values, 'a' to Add to existing
## value (use with tag-like columns), and 'n' for setting on New books
## only. (Default is 'r'.)

## Literal strings can be set into custom columns using double quotes.
## Each metadata=>column mapping must be on a separate line and each
## needs to have one space at the start of each line.

## 'r_anthaver' and 'n_anthaver' can be used to indicate the same as
## 'r' and 'n' for normal downloads, but to average the metadata for
## the differents story in an anthology before setting in integer and
## float type custom columns. This can be useful for a averrating
## column, for example. Default is to sum the values of all stories,
## and numChapters and numWords are always summed.

custom_columns_settings:
series01=>#series2
series02=>#series3
series03=>#series4
# cliches=>#acolumn
# themes=>#bcolumn,a
# timeline=>#ccolumn,n
# "FanFiction"=>#collection
# averrating=>#averrating,r_anthaver

[www.fanfiction.net]

## browser cache will only be used if use_browser_cache:true and ONLY
## for fanfiction.net right now.  Requires a browser_cache_path set in
## [defaults].
use_browser_cache:true

## use_browser_cache_only:true prevents FFF from falling through to
## request from the server when a page isn't in the browser cache.
## You can turn this on to be sure you're only using the browser
## cache.
use_browser_cache_only:true

## Pre-existing ffnet INI settings you might want to think about
## changing with use_browser_cache:

## If you want covers to work correctly with use_browser_cache, you'll
## need to click the cover image at least once for each story.  Or
## turn cover off.
never_make_cover:true

## If you want skip_author_cover to work correctly with
## use_browser_cache, you'll need to click the author page at least
## once for each story.  Or turn skip_author_cover off.
##
## From defaults.ini:
## April 2015, ffnet changed their story and user pictures urls to be
## the same.  Now the only way to know if the story image is really
## the author image is to go get the author image and check.
skip_author_cover:false

## If you turned this on before, you probably want it off while using
## browser cache.
##
## From defaults.ini:
## ffnet used to have a tendency to send out update notices in email
## before all their servers were showing the update on the first
## chapter.  It generates another server request and doesn't seem to
## be needed lately, so now default it to off.
check_next_chapter:true

[watani]

So a problem I've encountered when getting Ao3 fics from the browser cache is that for very large fics where only a few chapters have been added since the last update, if I have 'open_pages_in_browser' active FFF does not give enough time for the entire work to be loaded in the browser before claiming that the update was bad. I instead have to restart the update and then it succeeds without issue because the full work page has now been successfully added to the browser cache.

To be more specific, the navigate and first chapter are opened in the browser before the "loading metadata" popup goes away, the job progress goes up to ~98%, the browser starts to open the full work but before it finishes loading FFF wraps up and says there was a bad update. The specific error shown is the following:

Error https://archiveofourown.org/works/XXXXXXXX Browser Cache Failed to Load with error 'unpack requires a buffer of 4 bytes'

Would it be possible to add a longer delay to give the browser time to load these larger fics instead of FFF failing immediately?

[JimmXinu]

New Test Version Posted

2025-02-24
- Fix for BrowserCache for images--cache partitioned by parent(story) page.

So, this is something people have reported before--images not loading from browser cache well. I didn't understand what was going wrong before because I wasn't seeing it / not looking hard enough.

Now that I'm having to use BrowserCache for AO3 too, I saw it and figured out the issue. I think.

FYI, it didn't show up for ffnet because ffnet doesn't allow images except their own.

[JimmXinu]

Originally Posted by watani

So a problem I've encountered when getting Ao3 fics from the browser cache is that for very large fics where only a few chapters have been added since the last update, ...

Story URL so I can see it too would be really nice...

Yes, I could make it longer. But after opening in browser, it already tries reading the cache 5 times with sleeps in between before repeating the whole cycle. I'd like to have something more concrete than 'longer'.

[JimmXinu]

New Test Version Posted

2025-02-24
- BrowserCache: Better handle cache file changing/failing while reading.
- Fix for BrowserCache for images--cache partitioned by parent(story) page.

Originally Posted by JimmXinu

Story URL so I can see it too would be really nice...

Yes, I could make it longer. But after opening in browser, it already tries reading the cache 5 times with sleeps in between before repeating the whole cycle. I'd like to have something more concrete than 'longer'.

Of course, immediately after this, I see the same problem when I go download my latest update. Just to make me feel like a jerk, I'm sure.

This is less about waiting and more about the file not finishing writing before we read it. I added better error handling and bumped up the wait times on the last 2 sleeps.

[RoundRobbinReadz]

Originally Posted by JimmXinu

I'm not sure what you mean, but I think you assume a greater degree of FFF-Calibre integration than exists.

I am new to using both Calibre and FFF, and I am learning! FFF is pretty darn awesome, so that could be lending itself to this impression. I am impressed by everything that it can do! Thanks for making it! (I hope this is an ok place to say thank you).

Perhaps, I am meddling with powers I can't possibly comprehend! Probably just a part of learning something new. But I may be in over my head.

On that note, thanks for the link to the wiki! So much info is contained there. It’s fabulous. I don’t want to bug you on here every time I have a question; Especially if there’s a place I can study to learn before asking a question here.

Originally Posted by JimmXinu

I refer to the data FFF collects and handles as "metadata entries", which are separate from Calibre data, AKA columns. Simplifying a bit, FFF metadata entries are divided into 'standard' entries collected for most/all sites and 'site specific' entries that are only used by that site. For example, every site has title and author; snapeflavours is potionsandsnitches.org only.

Ok, great. This is very helpful. I think I get this concept.

Originally Posted by JimmXinu

If you want FFF to populate a Calibre custom column on download/update, you can set it in the FFF Custom Columns config tab (for standard metadata entries) in personal.ini with custom_columns_settings.

Yep, this is the thing I am working on. Sorry for not being more specific with my first question. I included more specifics below.

Originally Posted by JimmXinu

If you want a Calibre custom column to appear in your ebooks' title page on update (it's assumed the Calibre book is created during initial download), there's a checkbox in the FFF personal.ini config tab 'Pass Calibre Columns into FanFicFare on Update/Overwrite' and a button to show you want what FFF metadata entry names each column will be placed in. They will automatically be labeled the same as the Calibre Columns.

Yes! I love this feature! I found it 2 weeks ago and it made everything better. My columns are beautifully populated. Before, I was just trying to have a custom column show Ships. Felt like a manageable goal at the time, but I wasn’t able to make it happen. Selecting this checkbox in the FFF personal.ini config tab made it happen. Clicking the box for 'Pass Calibre Columns into FanFicFare on Update/Overwrite', changed everything! Gave me hope to try more things with FFF. Prior to this, I was trying to do things with metadata entries and basically seeing no results.

I want to figure out how to populate a custom column with either type of metadata entries, standard & site specific entries. I can probably figure this out from the wiki, now that I know where to look.

But I think part of where my problem is, and/or is indicated here:
Under the FFF config tab for Custom Columns, I see the name of the column I made on the left, but don't know what metadata value type to fill it with. The name of the column isn’t listed as an option in the menu. I presume this is related to it not being standard data entries, or if site specific, the personal.INI is incorrect.

However, it’s listed in the drop down menu under Special Column, Update/Override Error Column has several of my column names listed, even ones that seem to be populating?

Example: I have a column for Fandom, set to seek #Fandom, but the metadata value type I chose is Category. It seems to be working.

Originally Posted by JimmXinu

You can then used those metadata entry names in titlepage_entries or logpage_entries.

OK. I haven’t changed anything under the section for Titlepage_entries. Seemed like updating the Title Page was a good future goal.

Originally Posted by JimmXinu

The FFF github project has a wiki which also has an FAQ.

A lot of INI settings are documented primarily in defaults.ini. You can see the defaults.ini for FFF Calibre plugin from the button on the FFF personal.ini config tab , or on github.

Thank you for all of these! I am glad to discover the wiki.


I was trying the idea discussed in this thread. (Post numbers #8111, #10105 and #10106.)

https://www.mobileread.com/forums/sh...postcount=8111
https://www.mobileread.com/forums/sh...ostcount=10105

https://www.mobileread.com/forums/sh...ostcount=10106

This idea seemed smart. Have a column with all the Tags, and a column for Tags I edited (cleaned up) before I start learning anything Replace metadata. But, this may not be necessary at this time. I figured it was a good thing to explore next - cleaning up tags.
I tried working with this code, from #10106 under the archiveofourown section of personal INI. So far, I’m stuck. But now, I have the wiki to explore. It may hold the info I need to GROK!

[ChaoticAdventure]

Originally Posted by fishing_forever

I've been trying to use the workaround and set the browser_cache_path in defaults. It opens the fic link in my default browser (Firefox), but then I get this error:

HTTP Error in FFF 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)'(428)

I tried setting the browser_cache_age_limit to -1 but that didn't do anything.

When I try to update the same fic again with FFF, it doesn't open it in my browser a second time.

I am also having this issue with FireFox. I don't suppose you found a solution for it?

[Amalthia]

Originally Posted by ChaoticAdventure

I am also having this issue with FireFox. I don't suppose you found a solution for it?

I'm having similar problems when I try and update series. I don't know why but the caching from Ao3 doesn't seem to work on series.

I'm going to try again in a few days to update a few of the series I'm following.

I'm also not updating series every time there is an update. A few writers seem to have an update daily...which I guess I shouldn't complain about too much because at least they are updating their story.

[fishing_forever]

Originally Posted by ChaoticAdventure

I am also having this issue with FireFox. I don't suppose you found a solution for it?

Unfortunately not, but I found that if a fic is already in your Calibre library and there aren't additional chapters, you are able to update it. It will say "Already contains [number] chapters" if you select Update EPUB if New Chapters or "Update epub completed, added 0 chapters for [number] total" if you say Update EPUB Always.

However, the problem comes up if you try to overwrite an existing fic or if you are trying to download a new fic/new chapters of a fic.

[JimmXinu]

Originally Posted by RoundRobbinReadz

But I think part of where my problem is, and/or is indicated here:
Under the FFF config tab for Custom Columns, I see the name of the column I made on the left, but don't know what metadata value type to fill it with. The name of the column isn’t listed as an option in the menu. I presume this is related to it not being standard data entries, or if site specific, the personal.INI is incorrect.

Only the standard metadata entries are listed in the Custom Column dropdowns, and only for column types that make sense.

For example, a numeric column will only offer you Words and Chapters because those are the only ones guaranteed to be numbers.

Site specific metadata entries have to be set using custom_columns_settings.

Re: AO3 Issues (unrelated to RoundRobbinReadz' post)

I'm doing what I can, when I can, to address AO3's on-going issues.

Basically, their servers are having problems. Part of addressing that is more aggressive (cloudflare, etc) blocking of automated access. See https://archiveofourown.org/admin_posts/31204

It doesn't help that Mac (at least) has started blocking open_pages_in_browser for at least some users in a recent update. I don't have a fix for that as yet, and I don't know if there will be one.

Unfortunately, I don't have the bandwidth right now to help each individual user sort out their exact configuration issues with BrowserCache. Hopefully people can help each other.

[ownedbycats]

Is it strange that I haven't had any of the reported issues with AO3? Maybe just lucky.