[GUI Plugin] FanFictionDownLoader

[JimmXinu]

FYI, New version out with these changes:

- New site: ksarchive.com, thanks to Jade Aislin for implementing this.

- Fixes for mugglenet.com--added user/pass, better handling for summary and is_adult levels.

- Allow default_cover_image to use ${title}, etc; add cover_exclusion_regexp feature to exclude images from being covers.

As usual, there may be a delay before calibre notices the new version. You can get the latest by going in and telling calibre to install it again.

[cordy]

Originally Posted by JimmXinu

@cordy - I don't think you truly have the latest version of either calibre or the plugin.

The error you show at the end says calibre 0.8.40, not 0.8.43. But that shouldn't make any difference for this problem.

The plugin writes it's version to the calibre job log and has for a number of versions now. It should look like this:

Download FanFiction Book
FanFictionDownLoader v1.5.9
Adding jobs for URLs:

Plus, the line number in the plugin your exception post references is blank, again indicating version mismatch.

If your plugin version is that old, you're probably bumping into the format change ffnet made a while back.

I'm not familiar with portable installs, so I can't help you with that part.

Thanks for helping me... I battled against the plugin for 10 minutes and you where right, it was the very first version installed with the latest overwritten. In fact it was installed twice, and the worked against each other.

Now it's fixed, and your plugin works great on the portable calibre edition.

Thanks
Sam

[jackr]

Haven't gotten a response back from the site owner of Fimfiction.net after sending them a message a couple weeks ago; I will try sending another soon.

I know that the posts (if there are any) do get dated with a year, but that of course is sketchy for basing the year on. And the stories from the author's creation dashboard (or whatever is a good word for that) have a year on them for "Last Modified", but that doesn't help any if it is not there in the target html.

Oh well, it's ultimately a small thing. Anyways, keep up the awesome work!

[JimmXinu]

@jackr - I expect they'll start thinking about it when the site gets old enough that the story dates become ambiguous.

[cryzed]

Hey there JimmXinu, I just found out about your plugin for Calibre and it is really looking good so far ! The thing is, I wrote my own solution for FanFiction.Net and FictionPress.com quite a while ago, it's called Lemon.

After inspecting your adapter files for the respective sites I thought maybe you could use the two APIs I wrote for them: FanFiction.Net API and FictionPress.com API. They don't introduce any new dependencies besides html5lib (using html5lib in correlation with BeautifulSoup is a good idea anyhow) and even easily parse information that you are currently missing, e.g. the review count (which I would really like to see in the final file if possible).

And I realize, of course, that this is only personal preference but could you adapt the default template to something like this/this/this? The source for the templates used by Lemon can be found here and here. Only if you agree, of course, but quite honestly (no offense): I really think that the templates used by Lemon are an improvement over the ones currently used in the plugin.

Man, I sound like a smug asshole -- but this really isn't my intention, I just want to finally deprecate Lemon for something better and I would be all over your Calibre plugin if you made these small changes.

[JimmXinu]

@cryzed -

I'm not really inclined to change FFDL to use your APIs for ffnet and fpcom. If you were offering code for a new site, or significant new functionality, it would be a different matter.

As for review count, I don't see that the value is worth the effort. It's going to be out of date most of the time. FFDL has 25+ different adapters for different sites it would have to be added to. And if FFDL collects review count, why not the recs/favorites/hits counts that many sites have, too?

I've added it to the 'wishlist' of future features, but I don't see it happening any time soon.

(BTW, to counter the argument that review counts could be updated everyday: I strongly recommend everyone use email notices for new chapters, and to not poll all your stories every day for new chapters. It inflates hit counts and wastes bandwidth on the fanfic site. We can only do this as long as we don't annoy the sites enough to block us.)

As for changing the appearance of the output, looking at your screen shots, the only differences between Lemon and FFDL output I noticed are that Lemon doesn't put commas in the word count and it centers the TOC entries. And, predictably, I like the output from FFDL better--because it's the way I like it.

Have you looked at FFDL's personal.ini settings? Specifically, the titlepage_entries, include_tocpage, and output_css options? There's already a fair degree of customization available. (Fair warning--FFDL mobi output doesn't support output_css. In fact, FFDL mobi output is only one step above deprecated. I strongly recommend epub and conversion.)

If you have specific requests, I might be able to add some additional markup to the title and TOC page output for the CSS to influence.

Thanks for your interest.

[nomad062]

I'm playing around using Ubuntu and of course one of the first things I downloaded was Calibre - the second thing I wanted was your FFDL plugin. Awesome plugin. Saves me several steps now that I can do it in 1 shot.

At any rate when I went to download your plugin it states that I need Calibre 8.30 or later yet my Calibre says its 8.8. Any idea? or do I need to post something with Calibre?

[PeterT]

Do an install of calibre from the official web site, and not from the Ubuntu repository.

[nomad062]

Thanks for the info!

[JimmXinu]

What he said.

0.8.30 is specified because that's the earliest version I've tested it with. There's probably at least a few earlier versions it would work with, but 0.8.8 is too far back--there were significant changes to plugin support around 0.8.18.

[nomad062]

I guess I was misreading the 8.8 I thought it was more recent than 8.3 but its all fixed now so thank you both for the quick response

Laurin

[cryzed]

@JimmXinu all right, I did it myself then -- while I was it I also implemented support for special title entries that allow a degree of customization (e.g. br-tags).

I still think that your code in its current form is really bad: in nearly all your files you have unused partial or even full module imports (sometimes duplicated), nothing that resembles PEP-8, you do some really puzzling things with the encoding/decoding of UTF-8 strings in your code and rely on chardet to guess encodings at times (although I didn't discover where), you use BeautifulSoup but apparently have no clue how to navigate its structure, RegEx is a solution for everything in your code while only making things more complicated (e.g. using it to strip '#' at the end of a string instead of just splitting the string), many of your methods do unexpected things, you use decorators incorrectly (e.g. adapaters.adapter_fanfictionnnet.FanFictionNetSite Adapter.getAcceptDomains), you have many methods that have a "get"-prefix and simply return a constant or unchanged value... and that's only after looking at 3-4 files of your project. That's why I wanted you to use my API modules, many, if not all things, you currently do in the adapter for FanFiction.Net and FictionPress.com could have been simplified.

Things that have me stumped are also that somehow when adding new items to the validEntries list and a corresponding label to the titleLabels list in your base_writer.py code the label is not used like apparently intended in BaseStoryWriter.writeTitlePage and the "NO_TITLE_ENTRY" is used as a template, so one has to actually manually define a "<title_entry>_label" entry in the configuration files -- why do the "fall back labels" exist then in the first place? (I'm sure this one's my fault) Additionally modifying the EPUB_TOC* constants in the EpubWriter class somehow has no effect on the resulting table of contents that is actually created, why is that?

Of course, I realize that my code is surely not perfect either but reading your code is really hard and puzzling -- and that's not because the things you do are complicated: the overall structure is just bad. This time there's no way that I could not have sounded like an asshole, sorry.

[JimmXinu]

@cryzed - I'm not going to argue with you about it. I'm not a python expert, nor have I claimed to be. This is a hobby.

If you'd like to contribute any of your changes, please set up a clone to check your changes into and the other developers on the project and I will review them.

[jackr]

I was busy tweaking the look of the Book Details pane today when I learned about how links can be assigned to author names. If you are looking for any new ideas at the moment, having the author link direct to the authors page would be great.

[davidfor]

Originally Posted by jackr

I was busy tweaking the look of the Book Details pane today when I learned about how links can be assigned to author names. If you are looking for any new ideas at the moment, having the author link direct to the authors page would be great.

In the tag browser, right click on the "Authors" category and select "Manage Authors". The third column in the list is what you want. I don't know how long that has been there, but, I only discovered this in the last couple of weeks.