[GUI Plugin] FanFictionDownLoader

[JimmXinu]

This plugin has been replaced. Its replacement, FanFicFare contains all the same features plus new sites, features and fixes.

[Aleyst]

Thanks for this. It's pretty handy.

[JimmXinu]

Posting FanFictionDownLoader plugin 1.1.0, hoping to get some beta testers before pushing it out.

Internally, 1.1 is an almost complete re-write of the plugin from 1.0.

I was advised by more experienced plugin developers that the way I was handling background processing was not particularly safe. Given the number of unexpected crashes I kept getting working with 1.0, I saw their point.

1.1 uses Calibre's mechanism to launch separate background processes to download stories rather than using a background thread.

There is now a drop down menu from the plugin tool bar button for Add/Update/Configure/About.

Also, 'update' truly updates the book(s) selected now. Before the plugin gathered URLs from the selected books, but after that treated the URL as if the user had entered it. If the name or author on the book had been changed, either in Calibre or on the website, it would not update the existing book in Calibre, but treat it as a different book.

The plugin now downloads the stories, but does *not* add/update them in your library until all are downloaded and you confirm you want the updates. Again, this is because background processing updating the foreground is dangerous in Calibre.

There is (hopefully) better download error/update reporting now, but it could still use some more work.

There are more options now for how to update existing stories, whether by directly updating or by doing 'Add New' with a URL to a story that already exists in your library.

Under "Add New from URL(s)", there's an option to choose handling for "If Story Already Exists?". This is also sometimes called collision handling.

"Skip" and "Add New Book" should be self-explanatory.

"Update EPUB if New Chapters" and "Update EPUB Always" allow you to incrementally update a story you already have, but only if you use EPUB. "Update EPUB Always" will update the title page and metadata for the story even if there are no new chapters.

"Overwrite Always" will download the story and overwrite the chosen format. "Overwrite if Newer", only if the website story is newer.

"Update Calibre Metadata Only" does not download the story, only enough metadata to update the metadata Calibre keeps.

Update Existing's "Update Mode" is identical.

Under Configuration, there are some changes and new options.

With the split between "Add New" and "Update Existing", the options to take URLs from selected and prioritize selected vs clipboard are removed. There's still an option to take URLs from clipboard or not.

You can also make the menubar button always bring up "Add New" by unchecking "Default to Update when books selected?".

Jim

(Attachment removed after 1.1.1 posted.)

[Aleyst]

Sounds interesting, I'm just updating the main calibre program now, I'll update the plugin and have a mess around.

[JimmXinu]

1.1.1 contains everything from 1.1.0 plus:

• Fixes for problems found with Add New.
• Fix to 'delete other formats' when html.
• Hide 'update epub' options when format not epub.
• fanfic.castletv.net changed to castlefans.org/fanfic.
• Fix for twiwrite adult-only stories.

I think this is ready to be pushed out as the new version, but I'd appreciate it if I could get a tester or two to try it first.

Jim

[scoopaz]

Hi Jim,

Went to give you plugin a shot since I'm a big collector of fanfiction and use the web version extensively. When trying to download a fanfic from fanfiction.net I received this error message:

Traceback (most recent call last):
File "site-packages\calibre\gui2\__init__.py", line 340, in dispatch
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 544, in download_list_completed
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 589, in _add_or_update_book
File "site-packages\calibre\library\database2.py", line 3175, in create_book_entry
File "site-packages\calibre\library\database2.py", line 3171, in _add_newbook_tag
AttributeError: 'set' object has no attribute 'append'

I'm using the latest version of Calibe (0.8.34) on a Windows 7 machine.

Please help! I would love to have this plugin working.

Lisa

[JimmXinu]

Originally Posted by scoopaz

Went to give you plugin a shot since I'm a big collector of fanfiction and use the web version extensively. When trying to download a fanfic from fanfiction.net I received this error message:

I'm using the latest version of Calibe (0.8.34) on a Windows 7 machine.

Please help! I would love to have this plugin working.

@scoopaz, I'm downloading 0.8.34 now--I was still on 0.8.33.

What's the URL of the story that gave the error? Does it happen for all, or just one story?

Jim

[scoopaz]

Hi Jim,

thanks for the quick response!. The URL for the story I initially tried is :

http://www.fanfiction.net/s/7623520/1/The_Good_Brother

so I tried a few more on fanfiction.net, and got the same error. Went to www.whofic.com just to confirm with a generic (non-adult) story:

http://www.whofic.com/viewstory.php?sid=29654
and received the error again.

Lisa

[JimmXinu]

@scoopaz, I've figured it out.

It's a bug in Calibre, not the plugin.

There's a problem in the routine that creates new book entries with "Tags to apply when adding a book" set.

I'll report it in the calibre bug tracking system.

For now, if you remove your "Tags to apply when adding a book", it will work.

Even if you don't want to do that for regular use, please give it a try and confirm that it solves your problem.

Jim

[scoopaz]

Here's the data from the FFdownloader job:

Download FanFiction Book
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
FanFictionDownLoader v1.1.1
Logfile for book ID None (The Good Brother)
urlhttp://www.fanfiction.net/s/7623520/1/) idNone)
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
write to c:\users\scoopaz\appdata\local\temp\calibre_0.8.34 _tmp_1ijjwz\fanfictiondownloader_czx5j7\new-None-lzgitu.epub
File(c:\users\scoopaz\appdata\local\temp\calibre_0 .8.34_tmp_1ijjwz\fanfictiondownloader_czx5j7\new-None-lzgitu.epub) Updated(2012-01-08) more recently than Story(2012-01-07) - Skipping


Lisa

[scoopaz]

Hi Jim,

Just saw your post - removed my auto tag when importing and gave it another shot:

The good news: The ebook was created and imported into my library with no errors:

The bad news: I received a different error when trying to open the created epub:

calibre, version 0.8.34
ERROR: Could not open ebook: File is not a zip file

Traceback (most recent call last):
File "site-packages\calibre\gui2\viewer\main.py", line 64, in run
File "threading.py", line 483, in run
File "site-packages\calibre\ebooks\oeb\iterator.py", line 205, in __enter__
File "site-packages\calibre\customize\conversion.py", line 204, in __call__
File "site-packages\calibre\ebooks\epub\input.py", line 138, in convert
File "site-packages\calibre\utils\zipfile.py", line 751, in __init__
File "site-packages\calibre\utils\zipfile.py", line 786, in _GetContents
File "site-packages\calibre\utils\zipfile.py", line 801, in _RealGetContents
BadZipfile: File is not a zip file

In Calibre the size of the file shows as 0.00 kb

Lisa

[JimmXinu]

Originally Posted by scoopaz

Just saw your post - removed my auto tag when importing and gave it another shot:

The good news: The ebook was created and imported into my library with no errors:

The bad news: I received a different error when trying to open the created epub:

File "site-packages\calibre\utils\zipfile.py", line 801, in _RealGetContents
BadZipfile: File is not a zip file

In Calibre the size of the file shows as 0.00 kb

Lisa,

Well, that's not supposed to happen... But on the other hand, it is farther along and therefore progress.

Please try it again making sure that you're adding from the URL and have 'epub' and 'Add New' selected. If it still makes a zero length epub, please post the details from the Download job.

Thanks,

Jim

[scoopaz]

Hi Jim,

OK, Made sure it was set to add new ebook, and add new from URL and no go.

Here is the results from the job:

Download FanFiction Book
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
FanFictionDownLoader v1.1.1
Logfile for book ID None (The Good Brother)
urlhttp://www.fanfiction.net/s/7623520/1/) idNone)
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
write to c:\users\scoopaz\appdata\local\temp\calibre_0.8.34 _tmp_o2hwch\fanfictiondownloader_9vuddg\new-None-ivsafd.epub
File(c:\users\scoopaz\appdata\local\temp\calibre_0 .8.34_tmp_o2hwch\fanfictiondownloader_9vuddg\new-None-ivsafd.epub) Updated(2012-01-08) more recently than Story(2012-01-07) - Skipping

I tried uninstalling the Goodreads plugin and tried again with no luck. could something be wonky with my configuration?


Lisa

[kiwidude]

@scoopaz - the two Goodreads plugin messages are due to you having installed two plugins a long time ago that got superceded by the single Goodreads plugin. They have nothing to do with any behavior with the FanFictionDownloader plugin.

It is just calibre telling you that it is not possible to load those plugins (because they arent compatible with recent calibre versions, hence they were replaced with a single "Goodreads" plugin).

To get rid of the error messages, see the couple of posts here. You can either do it via the command line or as dwnathny says just delete the relevant plugin zip files from the calibre plugins folder. Just make sure you do the right ones - as per your error message text they are "Goodreads Covers" and "Goodreads Metadata" that are the plugins you should remove, and not the "Goodreads" one.

[JimmXinu]

And I was getting ready to blame it all on Goodreads plugins. :-)

Actually, Lisa, your last log posting gave me the answer.

I've replaced the zip file on the "FanFictionDownLoader 1.1.1" post above with a fixed one. Please give that one a try.

Thanks,

Jim