[GUI Plugin] FanFictionDownLoader

[JimmXinu]

@riothamus - What information are you trying to get FFDL to put in the custom column?

If it's one of the standard metadatum collected for stories from most sites, just select it for that column on the Custom Columns tab of FFDL's config.

[zraith]

Originally Posted by JimmXinu

Evidence continues to mount that ffnet has instituted some sort of throttling system that significantly slows down some requests once they decide that you've made too many. We don't know what their criteria are, but it does seem to be slow downs rather than outright blocking.

The only advice I can give is don't do 'too many' downloads or update at once, or in a day. The value of 'too many' is unknown, but I'd guess it's lower than 100.

I am curious as to the criteria for that throttling system as there isn't any reasonable rhythm to the throttling. It appears to be on the number of requests but then why would it also prevent downloading new stories that only have 1 chapter each (just pondering out loud, not expecting anyone to be able to answer this).

As for the number of requests, I'm seeing it throttle when trying to update an existing story that I have that has had ~30 new chapters added (~300 words each) since the last time I've updated it. What is weird is that just after stopping this job, I tried updating a story that only had 1 new chapter added and had no problem with updating it.

The other weird thing is that I've tried 2 other different services for downloading stories, 1 is a web service and while it took a while was able to download (https://www.fanfiction.net/s/8601250/1/A-Dead-World) without any problems. The other service was one that went by a similar name, Fanfictiondownloader (found at fanfictiondownloader.net), and was a stand-alone program that required to be installed on the PC. This 2nd one I tried it with 2 different stories, the aforementioned one and another bigger one that is over 500k words spread over 73 chapters, no problem with downloading them (for now I guess).

This is a little telling that it might have to be with how the FanFictionDownloader plugin is retrieving/requesting the stories off FFnet that is causing the problem.

On a side note is there a way to increase the timer between sending requests?

[riothamus]

Originally Posted by JimmXinu

@riothamus - What information are you trying to get FFDL to put in the custom column?

If it's one of the standard metadatum collected for stories from most sites, just select it for that column on the Custom Columns tab of FFDL's config.

I'm trying to populate a Collection column with the text "Fanfics" so they go to the Fanfics collection on the kindle. It's the only collection I have in this library.

[JimmXinu]

Originally Posted by zraith

...
The other weird thing is that I've tried 2 other different services for downloading stories, 1 is a web service and while it took a while was able to download (https://www.fanfiction.net/s/8601250/1/A-Dead-World) without any problems. The other service was one that went by a similar name, Fanfictiondownloader (found at fanfictiondownloader.net), and was a stand-alone program that required to be installed on the PC. This 2nd one I tried it with 2 different stories, the aforementioned one and another bigger one that is over 500k words spread over 73 chapters, no problem with downloading them (for now I guess).

fanfictiondownloader.net is unrelated to this project.

http://fanfictiondownloader.appspot.com/ is the web service version of this FFDL project.

Originally Posted by zraith

This is a little telling that it might have to be with how the FanFictionDownloader plugin is retrieving/requesting the stories off FFnet that is causing the problem.

It's possible the User-Agent used is an issue, as well. That can be configured in personal.ini with user_agent.

Originally Posted by zraith

On a side note is there a way to increase the timer between sending requests?

slow_down_sleep_time

[JimmXinu]

Originally Posted by riothamus

I'm trying to populate a Collection column with the text "Fanfics" so they go to the Fanfics collection on the kindle. It's the only collection I have in this library.

It's possible, but it's enough of a corner case that I've never make it easy. But since you're the third(?) person to ask for it in the last two years, let's change that.

Attached is a version that allows literal strings to be set using custom_columns_settings.

You have to have "Allow custom_columns_settings from personal.ini to override" checked on the Custom Columns tab and in personal.ini, add:

custom_columns_settings:
 "FanFics"=>#collection

The full spec is:

Spoiler:

## (Plugin Only) - You can also populate calibre custom columns with
## the site specific metadata using custom_columns_settings (but only
## if 'Allow custom_columns_settings from personal.ini' is checked in
## the plugin GUI config.)  There are three parts, the entry name,
## then the label of the calibre custom column, then (optionally) a
## 'mode'.  'r' to Replace any existing values, 'a' to Add to existing
## value (use with tag-like columns), and 'n' for setting on New books
## only.  (Default is 'r'.)
## Literal strings can be set into custom columns using double quotes.
## Each metadata=>column mapping must be on a separate line and each
## needs to have one space at the start of each line.

#custom_columns_settings:
# cliches=>#acolumn
# themes=>#bcolumn,a
# timeline=>#ccolumn,n
# "FanFiction"=>#collection

If this works well, it will be in the next release.

UPDATE Jan 21, 2014 - Remove obsolete beta versions

[riothamus]

Originally Posted by JimmXinu

It's possible, but it's enough of a corner case that I've never make it easy. But since you're the third(?) person to ask for it in the last two years, let's change that.

Attached is a version that allows literal strings to be set using custom_columns_settings.

You have to have "Allow custom_columns_settings from personal.ini to override" checked on the Custom Columns tab and in personal.ini, add:

custom_columns_settings:
 "FanFics"=>#collection

If this works well, it will be in the next release.

I tried this in several ways last night with no luck. I tried the following custom column types: Text show in tag browser and Text fixed values. I do have the checkbox marked on the Custom Column tab and the code in the personal.ini tab. I also made sure to install the attached version and restarted caliber. The column just refuses to update with anything.

[JimmXinu]

Originally Posted by riothamus

I tried this in several ways last night with no luck. I tried the following custom column types: Text show in tag browser and Text fixed values. I do have the checkbox marked on the Custom Column tab and the code in the personal.ini tab. I also made sure to install the attached version and restarted caliber. The column just refuses to update with anything.

Works fine for me with a "text show in tag browser" column.

What section of personal.ini did you add it to? [defaults]? Or a site specific section?

And you do understand that it won't be populated for a book until you download or update that book again?

[riothamus]

Originally Posted by JimmXinu

Works fine for me with a "text show in tag browser" column.

What section of personal.ini did you add it to? [defaults]? Or a site specific section?

And you do understand that it won't be populated for a book until you download or update that book again?

I do indeed realize it only populates when I add or update via the FFDL plugin.

I first put it in the overrides section at the bottom. I then moved it to just above the overrides section which might have been considered a site specific section. I just moved it to after all the default and before the start of the site specific settings and yep, that fixed it. I didn't realize the site specific sections worked like that.

Thanks! That works great. I only wish I'd asked over a year ago when I first started using the FFDL plugin. This and the Reading List plugin are the best plugins for my use of calibre.

[JimmXinu]

Version 1.8.08 - 21 Jan 2014

• Allow literal strings in custom_columns_settings to set columns.

[Firedancer885]

Originally Posted by JimmXinu

Version 1.8.08 - 21 Jan 2014

• Allow literal strings in custom_columns_settings to set columns.

Jimm, I didn't see the alert for the new version in Calibre so I downloaded the zip you put into the first post of the thread. I got the following error when I tried to install it:

calibre, version 1.20.0
HIBA: Ismeretlen hiba történt: <b>SyntaxError</b>:invalid token (calibre_plugins.fanfictiondownloader_plugin.__init__, line 45)

calibre 1.20  isfrozen: True is64bit: False
Linux-3.11.0-15-generic-i686-with-debian-wheezy-sid Linux ('32bit', 'ELF')
('Linux', '3.11.0-15-generic', '#23-Ubuntu SMP Mon Dec 9 18:16:27 UTC 2013')
Python 2.7.5
Linux: ('debian', 'wheezy/sid', '')
Traceback (most recent call last):
  File "site-packages/calibre/gui2/preferences/plugins.py", line 308, in add_plugin
  File "site-packages/calibre/customize/ui.py", line 375, in add_plugin
  File "site-packages/calibre/customize/ui.py", line 53, in load_plugin
  File "site-packages/calibre/customize/zipplugin.py", line 195, in load
  File "site-packages/calibre/customize/zipplugin.py", line 175, in load_module
  File "calibre_plugins.fanfictiondownloader_plugin.__init__", line 45
    version             = (1, 8, 08)
                                  ^
SyntaxError: invalid token

[JimmXinu]

Version 1.8.9 - 22 Jan 2014

• Leading zeros in version number makes it octal. Remove and bump to 1.8.9.

[JimmXinu]

Testing a fix for a typo in French translation.

UPDATE Jan 26, 2014 - Remove obsolete beta versions

[Etorak]

Hi,

I'm using Calibre in French. The last two version of the FanFictionDownLoader plugin didn't work for me. During the download phase, some errors would occurs.
However, with the version posted today everything works fine now. Not sure what you were trying to fix, but looks good to me !

[JimmXinu]

Originally Posted by Etorak

I'm using Calibre in French. The last two version of the FanFictionDownLoader plugin didn't work for me. During the download phase, some errors would occurs.
However, with the version posted today everything works fine now. Not sure what you were trying to fix, but looks good to me !

Sorry about that. That issue was reported today--it was a typo in the French translation file.

[julianabr]

I got some different errors when trying to update to 1.8.9 on my Mac.

When I tried to load by file:

calibre, version 1.20.0
ERROR: Unhandled exception: <b>InvalidPlugin</b>:The plugin in u'/Users/kindle/Downloads/FanFictionDownLoaderPlugin-1.zip' is invalid. It does not contain a top-level __init__.py file

calibre 1.20 isfrozen: True is64bit: True
Darwin-13.0.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '13.0.0', 'Darwin Kernel Version 13.0.0: Thu Sep 19 22:22:27 PDT 2013; root:xnu-2422.1.72~6/RELEASE_X86_64')
Python 2.7.5
OSX: ('10.9.1', ('', '', ''), 'x86_64')
Traceback (most recent call last):
File "site-packages/calibre/gui2/preferences/plugins.py", line 308, in add_plugin
File "site-packages/calibre/customize/ui.py", line 375, in add_plugin
File "site-packages/calibre/customize/ui.py", line 53, in load_plugin
File "site-packages/calibre/customize/zipplugin.py", line 188, in load
File "site-packages/calibre/customize/zipplugin.py", line 282, in _locate_code
InvalidPlugin: The plugin in u'/Users/kindle/Downloads/FanFictionDownLoaderPlugin-1.zip' is invalid. It does not contain a top-level __init__.py file


When I tried to use 'get new plugin' button:

calibre, version 1.20.0
ERROR: Install Plugin Failed: A problem occurred while installing this plugin. This plugin will now be uninstalled. Please post the error message in details below into the forum thread for this plugin and restart Calibre.

Traceback (most recent call last):
File "site-packages/calibre/gui2/dialogs/plugin_updater.py", line 711, in _install_clicked
File "site-packages/calibre/customize/ui.py", line 383, in add_plugin
OSError: [Errno 13] Permission denied: '/Users/kindle/Library/Preferences/calibre/plugins/FanFictionDownLoader.zip'