web2lrf

[kovidgoyal]

Yeah you'd have to figure out the arguments to web2disk that the BBC profile uses from the source code and pass them manually using the commandline.

[veshman]

I'm trying to write a converter for Wired magazine. I am totally new to python...how can I add the /print/ into the following URL?


http://www.wired.com/gadgets/digital...rning_question

http://www.wired.com/print/gadgets/digitalcameras/magazine/test2007/dc_burning_question

I'm thinking something like this might work....but I don't know how to make the latter part of the URL a variable that I can put back into the string.

return url.replace('wired.com/?', 'wired.com/print/?')

thanks,

bhavesh

[FixB]

Sorry veshman : I'm having the same difficulties here on some french rss
I would have thought your suggestion should work. Maybe you don't need the "?" as you just replace wired.com with wired.com/print ?
By the way, do someone know how I can keep (and access) the intermediate html files when using web2lrf, so that I could see exactly where my use of regular expressions is faulty ??

[FixB]

I tried and it seems that :

def print_version(self, url):
return url.replace('wired.com','wired.com/print')

works correctly.
But strangely, not for all articles. The first one seems ok, but the second one is in the 'normale' format... strange

[veshman]

perhaps it should be:

return url.replace('wired.com','wired.com/print/')

with a second "/"

I'll give it a try.

Also, any thoughts on how to keep web2lrf from pursuing external links (e.g. ads)?

thanks,

bhavesh

[veshman]

so I'm getting the URL to appear correctly using the url.replace function, but for some reason, web2lrf can't process the link.

Processing category6.html
Parsing HTML...
Converting to BBeB...
Could not follow link to http://www.wired.com/print/science/d...1/st_alphageek

If I just copy and paste the URL into a web browser, it works fine.

Bhavesh

[veshman]

using the url.replace code did work with the addtion of the "/" but web2lrf was unable to find the link, even though it created it correctly.

meaning, if I copy and paste the link that web2lrf is trying to get into a browser, it works fine.

[veshman]

on the exclude links front, I tried adding an operator to the script, but so far haven't figured it out.

link-exclude = [^wired]

or
link-exclude = ^w^i^r^e^d
or
link-exclude = *[^wired]*

and a number of other failed attempts that give me a syntax error.

[kovidgoyal]

I'm on my thanksgiving break right now, so I can't help in detail, but you may find this page helpful

http://docs.python.org/lib/re-syntax.html

[DaveNB]

Try this script. Copy the text below the ------ and save/paste it into a file called "wired.py", it'll produce a file:
Wired RSS [25 Nov 2007 1720].lrf (for example).

I think it's producing pretty clean text (most ads, links, banners, comments, cruft are removed) for reading off-line, but there are still some fomatting issues (some fonts too big, others way too small, maybe I need to kill all CSS info in the <header> sections completely?).

BTW, if you make any changes to the user profile wired.py file, before running the web2lrf command, delete the previously generated wired.pyc file or your changes won't be reflected (I think).

Any suggestions for cleaning up the text formatting? Give it a try.

Dave

-------

# coding: ISO-8859-1
##    Copyright (C) 2007 David Chen SonyReader<at>DaveChen<dot>org
##
##    This program is free software; you can redistribute it and/or modify
##    it under the terms of the GNU General Public License as published by
##    the Free Software Foundation; either version 2 of the License, or
##    (at your option) any later version.
##
##    This program is distributed in the hope that it will be useful,
##    but WITHOUT ANY WARRANTY; without even the implied warranty of
##    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
##    GNU General Public License for more details.
##
##	Version 0.6-2007-11-27
##	Based on newsweek.py, bbc.py, nytimes.py by Kovid Goyal
##	https://libprs500.kovidgoyal.net/wiki/UserProfiles
##
##	Usage:
##	>web2lrf --user-profile wired.py
##	Comment out the RSS feeds you don't want in the last section below
##
##	Output:
##	Wired [YearMonthDate Time].lrf
##
'''
Profile to download RSS News Feeds and Articles from Wired.com
'''

import re

from libprs500.ebooks.lrf.web.profiles import DefaultProfile 
	
class wired(DefaultProfile):
   
	title = 'Wired'
	max_recursions = 2
	timefmt  = ' [%Y%b%d  %H%M]'
	html_description = True
	no_stylesheets = True
	
	## Don't grab articles more than 7 days old
	oldest_article = 7
  
	preprocess_regexps = [(re.compile(i[0], re.IGNORECASE | re.DOTALL), i[1]) for i in 
		[

		## Remove any banners/links/ads/cruft before the body of the article.
		(r'<body.*?((<div id="article_body">)|(<div id="st-page-maincontent">)|(<div id="containermain">)|(<p class="ap-story-p">)|(<!-- img_nav -->))', lambda match: '<body><div>'),

		## Remove any links/ads/comments/cruft from the end of the body of the article.
		(r'((<!-- end article content -->)|(<div id="st-custom-afterpagecontent">)|(<p class="ap-story-p">&copy;)|(<div class="entry-footer">)|(<div id="see_also">)|(<p>Via <a href=)|(<div id="ss_nav">)).*?</html>', lambda match : '</div></body></html>'),

		## Correctly embed in-line images
		(r'<a.*?onclick.*?>.*?(<img .*?>)', lambda match: match.group(1),),

		## Correct the apostrophe character so it renders well in LRF
		(r'’', lambda match: "'"),
		]
	]

## Use the single page Print version of a page when available.
## Not all RSS entries have Print versions, ie. ones hosted on the blog.wired.com URL's

	def print_version(self, url):
		return url.replace('http://www.wired.com/', 'http://www.wired.com/print/')

## Comment out the feeds you don't want retrieved.
## Or add any new new RSS feed URL's here

	def get_feeds(self):
		return	[
		('Top News', 'http://feeds.wired.com/wired/index'),
		('Culture', 'http://feeds.wired.com/wired/culture'),
		('Software', 'http://feeds.wired.com/wired/software'),
		('Mac', 'http://feeds.feedburner.com/cultofmac/bFow'),
		('Gadgets', 'http://feeds.wired.com/wired/gadgets'),
		('Cars', 'http://feeds.wired.com/wired/cars'),
		('Entertainment', 'http://feeds.wired.com/wired/entertainment'),
		('Gaming', 'http://feeds.wired.com/wired/gaming'),
		('Science', 'http://feeds.wired.com/wired/science'),
		('Med Tech', 'http://feeds.wired.com/wired/medtech'),
		('Politics', 'http://feeds.wired.com/wired/politics'),
		('Tech Biz', 'http://feeds.wired.com/wired/techbiz'),
		('Commentary', 'http://feeds.wired.com/wired/commentary'),
		]

[veshman]

Dave,

thanks! i'll give it a try and post my results.

bhavesh

[veshman]

Kovid,

thanks for the link...it is very helpful. I'll try a couple of the expressions out.

bhavesh

[kovidgoyal]

version 0.4.25 finally implements support for The Economist. See demo attached to first post.

[DaveNB]

I edited the previous post to reflect the changes in the source code for the newest wired.py User Profile for web2lrf.

There is major improvement in the proper rendering/placement of inline images and proper display of inline hypertext linked phrases/words.

However, there are some issues with text encoding that to fix the problems with the apostrophe's (sometimes Wired uses a simple vertical tic, sometimes they use the apostrophe where the tail curves down to the left, the latter renders strangely as 3 international characters on the Sony Reader). Version 0.6 attempts to fix this but so far, I can't seem to get the right character/hex sequence for the problematic apostrophe character (right single quote) to substitute it out.

Give it a try and let me know if any one can figure out how to fix the apostrophe problem.

Dave

[kovidgoyal]

The problem with wired is that the files are encoded in UTF8 but they specify the encoding as iso8859-1. You can try either
1) Contact wired
2) write a preprocess regexp that changes the specified encoding

(r'<meta http-equiv="Content-Type" content="text/html; charset=(\S+)"',
 lambda match : match.group().replace(match.group(1), 'UTF-8'))