[GUI Plugin] FanFictionDownLoader

[Firedancer885]

Originally Posted by CyberCrisis

Anybody having login problem with www.tthfanfic.org.

In FFDL? I had. Read back a few posts, Jimm already solved it.

[Firedancer885]

Originally Posted by JimmXinu

Okay, in that case, make it just:

[www.tthfanfic.org]
user_agent:FFDL/1.7

Thanks

[Feynor]

If only these sites would make good use of the HEAD command, it could cut down on their load tremendously.
The standard response header does have a Last-Modified field, for instance:
Last-Modified: Tue, 10 Sep 2013 07:43:03 GMT

However on FFNET and TTH, this seems to always be today's date, rendering it useless.

Are there any way to "suggest" that they start using the response header to provide simple metadata?

Why would that help them?
By using the HEAD command, the server "only" have to look up a few things about the given story id, it doesn't have to load all the data for the page, and render the HTML code or send the data, taxing their bandwidth.

Why would that help us?
Speed, and easier decoding. Pure and simple.

[Sefiriot]

Originally Posted by Feynor

If only these sites would make good use of the HEAD command, it could cut down on their load tremendously.
The standard response header does have a Last-Modified field, for instance:
Last-Modified: Tue, 10 Sep 2013 07:43:03 GMT

However on FFNET and TTH, this seems to always be today's date, rendering it useless.

Are there any way to "suggest" that they start using the response header to provide simple metadata?

Why would that help them?
By using the HEAD command, the server "only" have to look up a few things about the given story id, it doesn't have to load all the data for the page, and render the HTML code or send the data, taxing their bandwidth.

Why would that help us?
Speed, and easier decoding. Pure and simple.

TTH might listen, no idea about FFN. Then again since they're currently trying to upgrade their network, this might be of use to their current interests. They've got a Twitter you can suggest this to.

[Feynor]

Originally Posted by Sefiriot

TTH might listen, no idea about FFN. Then again since they're currently trying to upgrade their network, this might be of use to their current interests. They've got a Twitter you can suggest this to.

Added the request to TtH: http://forum.tthfanfic.org/index.php?topic=8289.0
All I could find for FFN was their support email at http://www.fanfiction.net/support/
Twitter messages are not quite long enough to convey any meaningful technical message :P

[Sefiriot]

Originally Posted by Feynor

Added the request to TtH: http://forum.tthfanfic.org/index.php?topic=8289.0
All I could find for FFN was their support email at http://www.fanfiction.net/support/
Twitter messages are not quite long enough to convey any meaningful technical message :P

I forgot to mention that they've got a blog as well, and people have been leaving feedback there which the site devs actually respond to: http://blog.fictionpress.com. You might get a faster response there as opposed to emails. Their email inbox is pretty much swamped all the time.

[Feynor]

Originally Posted by Sefiriot

I forgot to mention that they've got a blog as well, and people have been leaving feedback there which the site devs actually respond to: http://blog.fictionpress.com. You might get a faster response there as opposed to emails. Their email inbox is pretty much swamped all the time.

Thanks.

[ClairePMR]

Originally Posted by Sefiriot

TTH might listen, no idea about FFN. Then again since they're currently trying to upgrade their network, this might be of use to their current interests. They've got a Twitter you can suggest this to.

Yes, and they actually replied to my request to consider adding their own epub download and compilation function with questions about what I would like to see. It's clear that they've never given this any thought while their competitors are, at the very least, allowing compilation for printing that allows saving as a full document. Many sites, AO3 and harrypotterfanfiction.com to name just two that I use, have added the ability to download as epub or send directly to a Kindle. FFN is missing the boat and is far behind other fan-sites on this.

BTW, I suggested that they google ffdl plugin to understand what I was requesting. That might get them to recognize that we're not attacking them, just trying to access their product in a way that they've not accommodated.

[JimmXinu]

Originally Posted by FaceDeer

...
So I went into adapter_fimfictionnet.py and figured out how to fix my own problem so it wouldn't be a hassle. I felt kind of embarrassed about how my previous bug report turned out to be a bug with Calibre itself rather than FFDL.
...

Looks like I got to it first--attached is a version that incorporates my version of your changes.

I've included a diff so you can see how I did it. I did use dateutil.parser like you did (stupid 2nd/3rd/4th issue), but I also used the same loop for publishedDate and updatedDate.

Spoiler:

@@ -21,7 +21,8 @@
 import re
 import urllib2
 import cookielib as cl
-from datetime import datetime
+#from datetime import datetime
+import dateutil.parser as dparser
 import json
 
 from .. import BeautifulSoup as bs
@@ -185,6 +186,7 @@
         hrstr="<hr />"
         descdivstr = '<div class="description">'+descdivstr[descdivstr.index(hrstr)+len(hrstr):]
         self.setDescription(self.url,descdivstr)
-        
+
+        # Can't trust dates from API anymore I'm told.
         # Dates are in Unix time
         # Take the publish date from the first chapter posted
@@ -189,10 +191,25 @@
         # Dates are in Unix time
         # Take the publish date from the first chapter posted
-        rawDatePublished = storyMetadata["chapters"][0]["date_modified"]
-        self.story.setMetadata("datePublished", datetime.fromtimestamp(rawDatePublished))
-        rawDateUpdated = storyMetadata["date_modified"]
-        self.story.setMetadata("dateUpdated", datetime.fromtimestamp(rawDateUpdated))
-
+        # rawDatePublished = storyMetadata["chapters"][0]["date_modified"]
+        # self.story.setMetadata("datePublished", datetime.fromtimestamp(rawDatePublished))
+        # rawDateUpdated = storyMetadata["date_modified"]
+        # self.story.setMetadata("dateUpdated", datetime.fromtimestamp(rawDateUpdated))
+        
+        oldestChapter = None
+        newestChapter = None
+        # Scan all chapters to find the oldest and newest, on
+        # FiMFiction it's possible for authors to insert new chapters
+        # out-of-order or change the dates of earlier ones by editing
+        # them--That WILL break epub update.
+        for chapterDate in soup.findAll('span', {'class':'date'}):
+            chapterDate = dparser.parse(chapterDate.contents[1].strip())
+            if oldestChapter == None or chapterDate < oldestChapter:
+                oldestChapter = chapterDate
+            if newestChapter == None or chapterDate > newestChapter:
+                newestChapter = chapterDate
+        self.story.setMetadata("datePublished", oldestChapter)
+        self.story.setMetadata("dateUpdated", newestChapter)
+        
         chars = soup.find("div", {"class":"inner_data"})
         # fimfic stopped putting the char name on or around the char
         # icon now for some reason.  Pull it from the image name with
@@ -215,7 +232,11 @@
                 if not isinstance(value,basestring):
                     value = unicode(value)
                 self.story.setMetadata(metakey, value)
-        
+
+        rawGroupList = soup.find('ul', {'id':'story_group_list'})
+        if rawGroupList is not None:
+            for groupName in rawGroupList.findAll('a', {'href':re.compile('^/group/')}):
+                self.story.addToList("groups",stripHTML(groupName))
             
     def getChapterText(self, url):
         logger.debug('Getting chapter text from: %s' % url)

UPDATE Sep 14 2013 - Remove obsolete beta versions

[silvrock]

Hi Jimm,

This is a post to just THANK YOU! I had suddenly realized that the FFDL plugin in my calibre was not working with fanfiction.net stories and my heart plummeted to my stomach. I checked the plugin updates and found you had released the updated plugin for calibre. I updated the plugin, restarted calibre and the fanfiction stories were properly downloading again and only then my heart found its way back to its proper place. And in that moment I thought someone ought to thank you on behalf of all the people who live by fanfics.
So, thank you.

Sending across lots of love,hugs, kisses, wonders and delight. Hope you find it intact in the package of this comment.


A fanfic reader, Silver.

[FaceDeer]

Originally Posted by JimmXinu

Looks like I got to it first--attached is a version that incorporates my version of your changes.

Aww, you beat me by like half a day. Tonight I had free time. Guess I'll use it testing, then.

Seems to be working correctly, I reran a bunch of updates and got the same results as my version. This code does break when trying to collect metadata for a story with *no* chapters (for example, this one) but that's not unexpected. FFDL already choked on those, it's just giving a different error message with this new version (previously such fanfics would fail to update with an error message that simply read "'chapters'"). It never even occurred to me to report this as a bug before, I have no idea what the correct behaviour should be for a case like this. Just thought I'd mention it since this version changes the previous behavior slightly.

I've got my own solution to detecting updated stories where authors have edited or inserted chapters earlier on. I have separate columns for both FiMFiction's *reported* word count and the word count calculated locally via the word count plugin, and a third column that compares the two. When a metadata update results in a significant difference between the two I check it out and see if it needs a forced overwrite to include edits. It's very rare, coming up with a system like that was more for fun than real necessity.

[seabream]

Originally Posted by cklls

It's clear that they've never given this any thought while their competitors are, at the very least, allowing compilation for printing that allows saving as a full document. Many sites, AO3 and harrypotterfanfiction.com to name just two that I use, have added the ability to download as epub or send directly to a Kindle. FFN is missing the boat and is far behind other fan-sites on this.

That might get them to recognize that we're not attacking them, just trying to access their product in a way that they've not accommodated.

I wouldn't be so sure that they've not given any thought to it. An issue is that FFN is an advertisement funded site whereas AO3 is donation funded. Using FFDL or hypothetically providing an epub as a whole work as opposed having each chapter separately on FFN webpages, drives/would drive down their ad views, which essentially means that people who are actually reading on the website are subsidising the people using FFDL because they actually provide revenue, while FFDL users just add to costs.

[ClairePMR]

Originally Posted by seabream

I wouldn't be so sure that they've not given any thought to it. An issue is that FFN is an advertisement funded site whereas AO3 is donation funded. Using FFDL or hypothetically providing an epub as a whole work as opposed having each chapter separately on FFN webpages, drives/would drive down their ad views, which essentially means that people who are actually reading on the website are subsidising the people using FFDL because they actually provide revenue, while FFDL users just add to costs.

Based on the direct communication I've had with them over the past couple of days, they in fact have not previously considered this technology. They asked numerous questions, most of which were focused on what the reader would need, but the nature of those questions made it clear that this was a new concept for them. While you're certainly right about the page views driving their revenues, they were very concerned about the fact that I was moving away to different sites because they didn't provide this functionality.

FFN was one of the first in the game, and while they have undoubtedly made some improvements in their technology and interface, they are one of the few fanfic websites that doesn't even allow automatic compilation for the purposes of printing. They are way behind the curve on that, and I think they should be concerned that readers will abandon them for other sources. As a fanfic author, I have posted there, but now post my work at three other sites that allow the reader to fully download either for printing or ebook compilation.

[Feynor]

I could probably even live with a small (unobtrusive) ad at the top of chapters, if it meant that we could sustain the functionality. How to do this though, without adding to the overhead of the ebook size I can't tell, not all downloaders support in-lined graphics if I recall.

[Sefiriot]

Originally Posted by Feynor

I could probably even live with a small (unobtrusive) ad at the top of chapters, if it meant that we could sustain the functionality. How to do this though, without adding to the overhead of the ebook size I can't tell, not all downloaders support in-lined graphics if I recall.

Not all reader apps either (well, not all handle them gracefully)-- and if reading with e-ink, graphics might not be the greatest idea, but since I've never used an e-ink device, others will be better able to judge. Still, to hear that FFN are actually considering compilation-- that's amazing! I should note however that there are authors out there who are against having their fics available in one easy download by default from a site, because of plagiarism fears. I know a person who refused to join AO3 precisely because of the availability of downloads, for the above reasons.