![]() |
#181 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,296
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
|
![]() |
![]() |
![]() |
#182 |
Addict
![]() ![]() ![]() ![]() Posts: 274
Karma: 332
Join Date: Nov 2003
Location: San Francisco, USA
Device: Sage, Elipsa, Oasis, Galaxy Tab 8U, S22U
|
Hi Kovid and all.
I looked at Atlantic and other profiles, seemed straightforward to parse the WSJ page. But knowing nothing about pyton doesn't help. Now I get to the point where it finds the links and downloads (I think it downloads), then I get this error: Traceback (most recent call last): File "convert_from.py", line 192, in <module> File "convert_from.py", line 186, in main File "convert_from.py", line 125, in process_profile File "libprs500\ebooks\lrf\web\profiles\__init__.py o", line 100, in __init__ File "libprs500\ebooks\lrf\web\profiles\__init__.py o", line 136, in build_inde x File "libprs500\ebooks\lrf\web\profiles\__init__.py o", line 115, in build_sub_ index KeyError: u'date' Here is the part that I changed: def parse_feeds(self): src = self.browser.open('http://online.wsj.com/page/2_0133.html').read() soup = BeautifulSoup(src) articles = [] for item in soup.findAll('a', attrs={'class':'bold80'}): url = item['href'] url = 'http://online.wsj.com'+url.replace('/article', '/article_print') title = self.tag_to_string(item) articles.append({ 'title':title, 'url':url, 'description':'' }) return {'Todays Paper' : articles } I didn't change the get_browser and preprocess_regexps, working fine in existing profile. Do you see anything obvious in my lines? I know not much info here to troubleshoot. I usually get one shot to run it in 2-3 hours. Because web2lrf doesn't log off from their site, next run cannot login for some time. How do you guys develop your profiles? Not much fun :-( Kovid, if you have nothing better to do and have time/desire to help me here, you have my login/password in your pm box, 2-3 weeks old. Just add "5" at the end of password, had to change at some point. Thanks in advance, David |
![]() |
![]() |
Advert | |
|
![]() |
#183 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,296
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
In the line articles.append You should have a 'date':time.time()
This will give all articles the default date. If you want the correct publication date you should parse the HTML for it. Note that you can define a cleanup function to logout. Something like Code:
def cleanup(self): self.browser.open('http://wsj.com/logout') Last edited by kovidgoyal; 02-22-2008 at 02:06 PM. |
![]() |
![]() |
![]() |
#184 |
Addict
![]() ![]() ![]() ![]() Posts: 274
Karma: 332
Join Date: Nov 2003
Location: San Francisco, USA
Device: Sage, Elipsa, Oasis, Galaxy Tab 8U, S22U
|
Thank you for quick reply!
I added the date, but cannot test it now because of their "security" policy :-( Not many logins allowed. I had added the same lines (from existing profile) for logout, but I cannot get it working. After creating the output, web2lrf does not exist (doesn't return to command prompt), just sits there: [INFO] convert_from.pyo:360: Converting to BBeB... [INFO] convert_from.pyo:283: Rationalizing font sizes... [INFO] convert_from.pyo:1754: Output written to C:\Misc\News\Wall Street Print Edition [Fri, Feb 22, 2008].lrf At this point I have to kill it. And WSJ doesn't like the next run. Without logging in, it simply creates an empty file because no articles are found. Thanks again for your help. I'll try again later. David |
![]() |
![]() |
![]() |
#185 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,296
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
I remember the WSJ website had some redirect nastiness that prevent web2lrf from logging out, there are some post about it earlier in this thread.
|
![]() |
![]() |
Advert | |
|
![]() |
#186 | |
Groupie
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 182
Karma: 1078201
Join Date: Sep 2007
Device: iPad Air 2
|
Quote:
With the current profile, I'm very careful about logging out of the site in my web browser before running it. And then I only run it once. Kind of a paid when you're testing changes in the profile, but I usually do most testing first without my login info. |
|
![]() |
![]() |
![]() |
#187 |
Addict
![]() ![]() ![]() ![]() Posts: 274
Karma: 332
Join Date: Nov 2003
Location: San Francisco, USA
Device: Sage, Elipsa, Oasis, Galaxy Tab 8U, S22U
|
You smarter guys couldn't fix it, and I didn't get any luck, so the same logout problem.
But with Kovid's help (and using profile from JTravers) I got my paper working. Now I'm getting all the articles from the print edition. It's not as nice as other profiles, simply lists all the articles by the page order (A1, A2..., B1, ..., etc.). Their feeds do not cover all articles from paper. Sometimes I start reading the paper in the morning, then leave for subway. Now I can continue reading the same article on the reader. |
![]() |
![]() |
![]() |
#188 | |
Groupie
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 182
Karma: 1078201
Join Date: Sep 2007
Device: iPad Air 2
|
Quote:
Thanks in advance! |
|
![]() |
![]() |
![]() |
#189 |
Addict
![]() ![]() ![]() ![]() Posts: 274
Karma: 332
Join Date: Nov 2003
Location: San Francisco, USA
Device: Sage, Elipsa, Oasis, Galaxy Tab 8U, S22U
|
I've used your profile, only changed the parsing part to this new parse_feeds.
I'm using the run time, didn't get the correct date from the page. Size is half of main WSJ profile (around 2Mb). Feel free to improve and post to libprc. Here is the method (I don't know how to post correctly, all indentation is gone): Code:
def parse_feeds(self): src = self.browser.open('http://online.wsj.com/page/2_0133.html').read() soup = BeautifulSoup(src) issue_date = time.ctime() articles = [] for item in soup.findAll('a', attrs={'class':'bold80'}): url = item['href'] url = 'http://online.wsj.com'+url.replace('/article', '/article_print') title = self.tag_to_string(item) articles.append({ 'title':title, 'url':url, 'description':'', 'date':issue_date }) return {'Todays Paper' : articles } |
![]() |
![]() |
![]() |
#190 | |
Groupie
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 182
Karma: 1078201
Join Date: Sep 2007
Device: iPad Air 2
|
Quote:
I'll take a look at it when I get some free time. |
|
![]() |
![]() |
![]() |
#191 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,296
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
I'm in the process of refactoring web2lrf to make it much more powerful and easier to use (impossible, I know). Here's an example of how it is more powerful, see the attached Newsweek ebook (downloaded using multi-threading in 10mins).
Feedback on the formatting and anything else is appreciated. |
![]() |
![]() |
![]() |
#192 |
Reticulator of Tharn
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 618
Karma: 400000
Join Date: Jan 2007
Location: EST
Device: Sony PRS-505
|
I really like the addition of the navigation panel at the beginning of each article, but do you think it would be possible for the 'next' link to come first, at least in the link-selection sequence? This might break the "traditional" order of navigation elements, but being able to skip through the articles with one button-press per article would greatly facilitate skimming (which at least for me would be the most common nav. panel use case).
|
![]() |
![]() |
![]() |
#193 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,296
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Yeah that's a good idea.
|
![]() |
![]() |
![]() |
#194 |
Addict
![]() ![]() ![]() ![]() Posts: 274
Karma: 332
Join Date: Nov 2003
Location: San Francisco, USA
Device: Sage, Elipsa, Oasis, Galaxy Tab 8U, S22U
|
This looks great.
And I like the idea for the "next" link. Now the question: when it will be ready? :-) |
![]() |
![]() |
![]() |
#195 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,296
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Well the code for it has already been committed to svn. Just needs testing, integration. It probably wont reach the GUI for at least a couple more releases.
|
![]() |
![]() |
![]() |
Tags |
libprs500, web2lrf |
Thread Tools | Search this Thread |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
web2lrf to capture blog archive? | Deputy-Dawg | Sony Reader Dev Corner | 1 | 02-14-2008 11:41 PM |
web2lrf: La Repubblica | alexxxm | Sony Reader | 1 | 11-13-2007 12:27 PM |