[GUI Plugin] FanFictionDownLoader

[paegan]

Almost forgot...
Dunno if this falls within the scope of your plugin but as an additional feature for the book id's if it would be possible to use customised ids to reference fanfic sites e.g.

http://www.fanfiction.net/s/5149431/...han_Fanfiction
would be something like:
ffn:5149431

or http://ficwad.com/story/181839
would be fwad:181839

Hopefully you get the idea I'm tryin to put accross

[JimmXinu]

Originally Posted by paegan

http://www.fanfiction.net/s/5149431/...han_Fanfiction
would be something like:
ffn:5149431

or http://ficwad.com/story/181839
would be fwad:181839

Hopefully you get the idea I'm tryin to put accross

Meaning you want to see those somewhere? Because siteabbrev and storyId are already a couple of the metadata fields the plugin makes available.

If you're asking to be able to enter those instead of URLs, I suppose it's possible, but I don't see the point--it's a lot easier, IMO, to copy or drag the URL.

[paegan]

Originally Posted by JimmXinu

Meaning you want to see those somewhere? Because siteabbrev and storyId are already a couple of the metadata fields the plugin makes available.

If you're asking to be able to enter those instead of URLs, I suppose it's possible, but I don't see the point--it's a lot easier, IMO, to copy or drag the URL.

Hmmm tried it... dosent work exactly like I hoped it would but meh, it's wasn't a necessity just a nice bit of polish (although if some1 else writes a plugin that does do that I do hope that you'd support it )

And to toss another feature idea @ you (which kinda obliquely relates to the custom ids thing):

Story mirrors i.e. the same story from multiple sources eg.
http://ficwad.com/story/182760 and
http://www.fanfiction.net/s/7820904/...Life_b_bBond_b

Now feel free to tell me where to get off cuz I kno this would be insanely difficult to implement eg. having to deal with choosing which link to update from, which like to crawl for metadata, etc. etc.

Nothing ventured, nothing gained ...

[JimmXinu]

Originally Posted by paegan

Hmmm tried it... dosent work exactly like I hoped it would but meh, it's wasn't a necessity just a nice bit of polish (although if some1 else writes a plugin that does do that I do hope that you'd support it )

I still don't understand what it is you want here. If you want 'ffnet:1324' to appear in calibre, put siteabbrev and storyId each in a custom column and use a third composite column to display them together. I use that to have a wordcount/chapters column, like: 12,123/7.

Originally Posted by paegan

Story mirrors i.e. the same story from multiple sources eg.
http://ficwad.com/story/182760 and
http://www.fanfiction.net/s/7820904/...Life_b_bBond_b

I don't see that there's anything particularly useful to be done there. I've seen the same story on different sites with different names, or chaptered differently, or different author name, etc... You can't make any useful assumptions.

[JimmXinu]

New Version up with support for a new site: gayauthors.org. Thanks to Ida Leter for implementing this.

[JimmXinu]

Testing for 1.5 is finished. It's been released for general use.

These changes are all for EPUB format ebooks.

The big changes are:

• Support for downloading images in the story text.
• Support for cover image from three different sources.
• Support to resize, make grayscale, and convert images to jpg.
• Option for plugin to update calibre cover image on metadata update.
• Major re-write of EPUB update to support images.
• Improved handling of ini section priority.
• New 'Stay on Top' option for 'Add New from URL(s)' dialog for easier drag-n-drop.
• Add replace_hr option to replace <hr> tags with '* * *'
• Support for fimfiction.net's locked story passwords.

Images
There's several new config parameters you can add to your personal.ini to turn on these features.

All of them should go under the [epub] section or the epub section for individual sites, such as [www.ficwad.com:epub].

The first turns images on in general. Without this, no images will be collected and it defaults to false.

include_images:true

The next allows the summary section in the title page to preserve its HTML from the source site. That way, images in the summary will be included. More on that in a moment. Summary metadata given to calibre still has HTML stripped for consistency.

keep_summary_html:true

As a convenience, and the draw the attention of more casual users, I added an 'Include images in EPUBs' checkbox to the Basic config tab. This is equivalent to adding:

[defaults]
include_images:true
keep_summary_html:true
make_firstimage_cover:true

...to the top of personal.ini. Your settings in personal.ini will override this.

Covers
There are three places cover images can come from.

First is an explicit cover image from the source site. Today, fimfiction.net is the only one--please let me know (with example URLs) if any of the other sites do, too.

Second, the downloader can use the first image it finds in the story with this paramter. If keep_summary_html is also true, it can come from the summary.

make_firstimage_cover:true

Third, you can set a specific image that will be used if one of the other two isn't found.

[www.adastrafanfic.com:epub]
default_cover_image:file:///C:/Users/username/Desktop/nook/images/icon.png
[www.fictionalley.org:epub]
default_cover_image:http://www.somesite.com/someimage.gif

Finally, if you want images inside the EPUB, but never want an image cover, you can use:

never_make_cover:true

Image options
All images saved to EPUB are converted to jpeg to save size. Further, you can set what size to scale the images down to fit in. Aspect ratio is preserved and images are never scaled up, only down. You can also have them converted to grayscale for additional(minor) space savings.

image_max_size: 580, 725
grayscale_images: true

The default size (580,725--width,height) is for my nook STR. If people tell me what appropriate sizes are for other book readers, I'll include them in the comments.

calibre cover
The plugin has a new option on the Basic tab, "Update Cover when Updating Metadata". When checked, the calibre cover image will be updated everytime the other calibre metadata is updated (assuming EPUB). If there is a cover image in the EPUB, it will be used. If not, calibre will generate the usual 'from EPUB format' type cover, usually showing the title page.

Internal changes
The code for updating existing EPUBs to add new chapters has been completely rewritten. You shouldn't see any change from the outside. Other than supporting images, that is.

Priority of personal.ini sections has been more formalized. When downloading an EPUB from ficwad, the sections will take priority in this order, from least to most specific:

1. [defaults]
2. [www.ficwad.com]
3. [epub]
4. [www.ficwad.com:epub]
5. [overrides]

Which is why, if you set options in [epub], to override them for ficwad, you have to set them in [www.ficwad.com:epub]

For convenience of cutting and pasting, here's all the new image parameters together:

Spoiler:

[epub]
## include images from img tags in the body and summary of
## stories.  Images will be converted to jpg for size if possible.
include_images:true

## If not set, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
keep_summary_html:true

## If set, the first image found will be made the cover image.  If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
make_firstimage_cover: true

## If set, the epub will never have a cover, even include_images is on
## and the site has specific cover images.
#never_make_cover: false

## If set, and there isn't already a cover image from the adapter or
## from make_firstimage_cover, this image will be made the cover.
## It can be either a 'file:' or 'http:' url.
## Note that if you enable make_firstimage_cover in [epub], but want
## to use default_cover_image for a specific site, use the site:format
## section, for example: [www.ficwad.com:epub]
#default_cover_image:file:///C:/Users/username/Desktop/nook/images/icon.png
#default_cover_image:http://www.somesite.com/someimage.gif

## Resize images down to width, height, preserving aspect ratio.
## Nook size, with margin.
image_max_size: 580, 725

## Change image to grayscale, if graphics library allows, to save
## space.
#grayscale_images: true

fimfiction.net locked story passwords
The downloader will now prompt for password on fimfiction.net's locked stories. Because each locked story has a different password, it's not useful to store a password in personal.ini. You can, however, use the fail_on_password option to prevent the plugin from prompting for story passwords.

[www.fimfiction.net]
fail_on_password: true

I appreciate hearing from both people with problems and people without, thanks.

v1.5.1 adds error handing for image loading, stay-on-top
v1.5.2 fixes a problem d/l'ing chapter text in GA.
v1.5.3 fixes a fimficnet problem, addes replace_hr & never_make_cover options.
v1.5.4 more fimfic fixs
v1.5.5 fimfic story password support & fail_on_password option
v1.5.6 add 'Include images in EPUBs' checkbox
v1.5.7 add 'nook_img_fix' - wrap img tags with div if not already wrapped with div or p for a nook problem. Defaults to true.

Jim

[jackr]

I am very excited with these new updates . I am trying the beta out and so far everything seems to work except for when I use "Update Calibre Metadata Only". When I use that option, the cover gets overwritten with a screen-shot of the summary page.

And I noticed that it is set to inject the first image found into the epub, which I like the idea of for sending it to the Nook w/o having to do the reconvert method. But I think having an option as to whether the epub gets injected with the cover or not would be a good idea, if it is possible*

That's one thing I never understood about covers when I was working on an ebub in Sigil once. I was able to assign a cover image and keep it from being within the html pages. But it wasn't kosher, even though the Nook displayed the cover just fine.

[JimmXinu]

Originally Posted by jackr

I am very excited with these new updates . I am trying the beta out and so far everything seems to work except for when I use "Update Calibre Metadata Only". When I use that option, the cover gets overwritten with a screen-shot of the summary page.

Hmm. I think that's more a matter of labeling than an actual problem. The 'Update Cover' option tells calibre to take the cover from the epub. If the epub hasn't been updated and doesn't have a cover image, it does the usual default. Finding the 'first image' would require downloading the whole story again in the case of a story with no images--not exactly what we want for 'update calibre only'.

Originally Posted by jackr

And I noticed that it is set to inject the first image found into the epub, which I like the idea of for sending it to the Nook w/o having to do the reconvert method. But I think having an option as to whether the epub gets injected with the cover or not would be a good idea, if it is possible*

If you have make_firstimage_cover off and don't have default_cover_image set, the epub won't have a cover in it. Is that sufficient, or do you mean something else?

Originally Posted by jackr

That's one thing I never understood about covers when I was working on an epub in Sigil once. I was able to assign a cover image and keep it from being within the html pages. But it wasn't kosher, even though the Nook displayed the cover just fine.

Unfortunately, covers are not specified by the epub standards, so it's a ugly as different readers have different ideas. What I've implemented follows the 'best practices' I've been able to find.

Thanks for the feedback, it's very helpful!

[jackr]

Originally Posted by JimmXinu

If you have make_firstimage_cover off and don't have default_cover_image set, the epub won't have a cover in it. Is that sufficient, or do you mean something else?

I tried removing make_firstimage_cover from personal.ini, but a cover is still injected into the epub.

What I mean about the cover is having it as an option so that only Calibre updates the cover. Before when I downloaded a new fanfiction and I either generated a new cover or pasted in an image, that cover wasn't injected into the epub. But maybe that is what you mean and I just can't get it to work yet .

Once again though, amazing work . If I could send more karma pts, I would have. I will go hunting for some stories with images in them to test next.

[JimmXinu]

Originally Posted by jackr

I tried removing make_firstimage_cover from personal.ini, but a cover is still injected into the epub.

Is it a story from fimfiction.net? That site has specific cover images, so it gets set whenever the cover image exists and include_images is on. That is a case where a 'don't make a cover' option would be useful, I suppose.

If not, can you give me the story URL? And maybe a copy of your personal.ini--minus user/passwords, of course.

Originally Posted by jackr

Once again though, amazing work . If I could send more karma pts, I would have. I will go hunting for some stories with images in them to test next.

Thanks, glad you like it.

[jackr]

Yes it was fimfiction stories that I tested with. My personal.ini is just this:

[defaults]
is_adult:true
#Accept-Encoding=gzip

[epub]
include_images:true
keep_summary_html:true
make_firstimage_cover:true

I am not to clear on whether accept-encoding needs a colon or equals, so I have it on standby for now.

An idea that just popped in my head as a possibility for a custom column option is an error code or some sort for stories that have errors when updating. I've come across a couple that no longer exist, and just a bit ago one that had some message about news or some sort for it (its error icon was the red one with a white x, I will check later to see what the exact error was).

Edit:
The error code I got was:

local variable 'newsrc' referenced before assignment

for http://www.fimfiction.net/story/5947/.

Another error code I've gotten is:

list index out of range

for http://www.fimfiction.net/story/9948/.

As well as several:

'NoneType' object has no attribute 'find'

for fimfiction stories that have either disappeared from the site or have had password locks placed on them by their authors.

[JimmXinu]

@jackr -

Re: cover image--the plugin gives fimfiction stories a cover whenever story has a cover image on the site and include_images is true. I'll add an option to suppress cover generation in general for such cases.

Re: Accept-Encoding--there's no such option in personal.ini, you can delete it.

Re: http://www.fimfiction.net/story/5947/ --downloaded fine for me with v.1.5.2.

Re: http://www.fimfiction.net/story/9948/ --Failed for me, too. I'll work on it.

Re: disappeared from the site or have password locks --can you give me an example URL for each?

Thanks for your feedback, it's very helpful.

[JimmXinu]

Originally Posted by JimmXinu

Re: http://www.fimfiction.net/story/9948/ --Failed for me, too. I'll work on it.

Failed because there's no chapter in that story--it reports 0 words, too. I'll fix the downloader so it doesn't fail outright, but I'm not sure downloading an empty story is useful.

[JimmXinu]

Beta version is now up to v1.5.3:

https://www.mobileread.com/forums/sho...85#post1982785

v1.5.1 adds error handing for image loading, stay-on-top
v1.5.2 fixes a problem d/l'ing chapter text in GA.
v1.5.3 fixes a fimficnet problem, adds replace_hr & never_make_cover options.

Hopefully this is just about ready to go.

[jackr]

I didn't notice the update to the beta, as well as the empty story (I just loaded it and noticed it had an image, didn't look at chapter list) X-).

Where is Accept-Encoding? I know you mentioned this feature a while back, but I couldn't find it elsewhere in the plugin. I probably misread your post about it .

http://www.fimfiction.net/story/9439/ is an example for a story that no longer exists.
And http://www.fimfiction.net/story/6884/ for a password locked story.
Both of these get the 'NoneType' object error.

I ran the plugin (v.1.5.3), and came up with a lot more errors than in 1.5. A majority of these new errors are:

local variable 'datePublished' referenced before assignment

Also the story that you mentioned worked in 1.5.2 has en error for me in 1.5.3. (http://www.fimfiction.net/story/5947/)

I've uploaded an image showing all the errors I got this time around. http://i39.tinypic.com/16k57o4.png