Custom recipes (archive, read-only)

[TonytheBookworm]

Originally Posted by Starson17

Possibly:


You probably know this, but just in case, the "difference between #bd and .bd" is that the first is for id="bd" and the second for class="bd". Usually, a class is for multiple elements on a page, while id is for individual elements, although that general rule is observed mostly in the breach.

No i did not know the difference between the # and the . that is what i was kinda wondering in my question actually. If you had to specify the tag name or the id name or the class name and what the syntax was.

Spoiler:

I think i'm correct in saying if i wanted to do h1 tags i would simply put in the extra_css
....
h1{whatever goes here}
then if i had say <div id=test1>.... </div>
extra_css
......
#test1{whatever goes here}

then if i had say <div class=test2>.....</div>
extra_css
........
.test2{whatever goes here}

[Starson17]

Originally Posted by TonytheBookworm

Yeah, sorry about the typo but i tried that and still did not work.
I have noticed that in some cases the author will use a .headline
or simply type h1 or even #div

You may need to remove styles first. Calibre will not strip them unless you tell it to in your recipe

no_stylesheets        = True
or
    def preprocess_html(self, soup):
        for item in soup.findAll(style=True):
            del item['style']
        return soup

h1 would be all <h1>, h1.bd in the CSS would be <h1 class="bd">, .bd would be all class="bd", etc. Look at the CSS file in your temp directory (after using ebook-convert .....) to see what happened. Open the first index.html in the temp directory, browse down to the articles, and learn to use the CSS explorer in FireBug. I know you use it for spotting the code on the web, but the CSS is useful, too. You can dynamically change the CSS in your local copy of the downloaded articles/recipe by opening it in html form in FireFox with FireBug and using FireBug directly on your downloaded html, instead of on the original article on the web. It's useful in both spots! Play with the CSS and your article changes dynamically.

And a side note. Man I have noticed after learning from you. I have been making recipes left and right and all for myself and for others. Never thought I would be doing that but none the less thanks for teaching me.

I noticed. That's great! It's exactly what I hoped would happen. It's more fun to teach to fish than give a fish
I suck at CSS, too, but I get by, and most of what I've learned has come from others who do more format conversions than I do.

[eldan]

Hi Starson17,

thanks for your reply.

Originally Posted by Starson17

1) That error indicates your login credentials are not being sent correctly. Can you login with those credentials manually?

Yes, I can login on that site without a problem. My password contains a lot of special chars. Maybe they need to be escaped in some way (urlencode)?

Originally Posted by Starson17

2) The name of the recipe is the same as the name you used to select it for normal use/download. Choose the pull down arrow next to News and select "Add a custom news source" then "Customize builtin recipe" then choose the name to add it to the custom list, then select it in the list to see the code.

I have found the script. Is there any way to access the script on the file system? I could not find the script in /Applications/calibre.app/Contents/Resources/resources/recipes/

[Starson17]

Originally Posted by eldan

Yes, I can login on that site without a problem. My password contains a lot of special chars. Maybe they need to be escaped in some way (urlencode)?

The recipe doesn't directly handle the password. It gets it as stored by Calibre. You can try changing your password to something less difficult (for testing), or modifying the recipe to process/escape it. I often revise a recipe to say passwords are not necessary, then define the user/pass internally and use them anyway. You may also want to see what is sent for login by using TamperData and FireFox. Perhaps you can store the password escaped.

I have found the script. Is there any way to access the script on the file system? I could not find the script in /Applications/calibre.app/Contents/Resources/resources/recipes/

It's stored in metadata.db. SQLiteSpy can get it, but why bother? I just copy it from the custom page (where I sent you) and paste it into my editor.

[TonytheBookworm]

Starson17,
Do you mind looking at this when you get a sec and telling me what the heck I'm doing wrong as far as the css is concerned please?
What my objective is, is to change this

Egypt’s housing market recovers!

it has the tag format of <div class="cdmainarticle">Egypt’s housing market recovers!</div>
So based on what I have gathered from other recipes and from you my code "should reformat it", but it doesn't.

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe

class GlobalProperty(BasicNewsRecipe):
    title = 'Global Property Guide'
    language = 'en'
    __author__ = 'TonytheBookworm'
    description = 'This is a site for residential property investors who want to buy houses or apartments in other countries'
    publisher = 'GlobalPropertyGuide.com'
    category = 'prices,real-estate'
    oldest_article = 10
    max_articles_per_feed = 100
    no_stylesheets = True
    extra_css = 'div.cdmainarticle{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;}'
    
    keep_only_tags    = [
                         dict(name='div', attrs={'class':['cd_mainbody']})
                        ]
    remove_tags = [
                   dict(name='div', attrs={'class':['addthis_toolbox addthis_default_style']}),
                   
                  ]                    
    feeds          = [
                      ('Main Feed', 'http://www.globalpropertyguide.com/rss'),
                      
                    ]

[Starson17]

Originally Posted by TonytheBookworm

Starson17,
Do you mind looking at this when you get a sec and telling me what the heck I'm doing wrong as far as the css is concerned please?
What my objective is, is to change this

Egypt’s housing market recovers!

it has the tag format of <div class="cdmainarticle">Egypt’s housing market recovers!</div>
So based on what I have gathered from other recipes and from you my code "should reformat it", but it doesn't.

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe

class GlobalProperty(BasicNewsRecipe):
    title = 'Global Property Guide'
    language = 'en'
    __author__ = 'TonytheBookworm'
    description = 'This is a site for residential property investors who want to buy houses or apartments in other countries'
    publisher = 'GlobalPropertyGuide.com'
    category = 'prices,real-estate'
    oldest_article = 10
    max_articles_per_feed = 100
    no_stylesheets = True
    extra_css = 'div.cdmainarticle{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;}'
    
    keep_only_tags    = [
                         dict(name='div', attrs={'class':['cd_mainbody']})
                        ]
    remove_tags = [
                   dict(name='div', attrs={'class':['addthis_toolbox addthis_default_style']}),
                   
                  ]                    
    feeds          = [
                      ('Main Feed', 'http://www.globalpropertyguide.com/rss'),
                      
                    ]

1) It has class="cd_mainarticle", not class="cdmainarticle",
2) It has inline style on your header. Strip that first:

    def preprocess_html(self, soup):
        for item in soup.findAll(attrs={'style':True}):
            del item['style']
        return soup

Try this one:

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe

class GlobalProperty(BasicNewsRecipe):
    title = 'Global Property Guide'
    language = 'en'
    __author__ = 'TonytheBookworm, with a little help from his friends'
    description = 'This is a site for residential property investors who want to buy houses or apartments in other countries'
    publisher = 'GlobalPropertyGuide.com'
    category = 'prices,real-estate'
    oldest_article = 10
    max_articles_per_feed = 100
    no_stylesheets = True
    extra_css = '''
                 .cd_mainarticle{font-family:Arial,Helvetica,sans-serif; color:red; font-weight:bold;font-size:large;}
                 '''    
    keep_only_tags    = [
                         dict(name='div', attrs={'class':['cd_mainbody']})
                        ]
    remove_tags = [
                   dict(name='div', attrs={'class':['addthis_toolbox addthis_default_style']}),
                   
                  ]                    
    feeds          = [
                      ('Main Feed', 'http://www.globalpropertyguide.com/rss'),
                      
                    ]
    def preprocess_html(self, soup):
        for item in soup.findAll(attrs={'style':True}):
            del item['style']
        return soup

Edit: (I made the header red to spot the change easily.)

[Thetasquared]

I would like to request a recipe for the "Current Issue" of Science News.

The rss feed is:
http://www.sciencenews.org/view/feed...ame/issues.rss

I know that a Science News recipe exists, but simply switching the feeds does not return articles for the "current issue"

[Starson17]

Originally Posted by Thetasquared

I would like to request a recipe for the "Current Issue" of Science News.

The rss feed is:
http://www.sciencenews.org/view/feed...ame/issues.rss

I know that a Science News recipe exists, but simply switching the feeds does not return articles for the "current issue"

That is not a feed for the "current issue", and it doesn't point to articles of the current issue. It's a feed for multiple issues and each feed, instead of pointing to a typical article, points to a web page with links to the articles in that issue.

Are you looking for one issue and its articles, or all the issues in that feed and all the articles in them?

It looks interesting, so I may tackle it.

[krunk]

@Starson17

I've modified your recipe, made a few blog additions, etc. and created a daily blog roll recipe.

Only problem is, it looks like the extra_css properties are not being applied.

If I:

ebook-convert my.recipe output && grep -ir "astyle" output/

To dump the OEB then grep for one of the custom styles, nothing is found. The documentation says extra_css should be placed in the <style></style> tags of the <head>. So I checked in output/index.html to be sure and nothing was placed their either.

I tried a few things like simplifying the extra_css and removing the no_stylesheets.

The modified recipe looks like:

# vim:ft=python

__license__ = 'GPL v3'
__copyright__ = '2010, James Kyle <james at jameskyle dot org>'

"""
Daily Blog Roll
"""

from calibre.web.feeds.recipes import BasicNewsRecipe
import re

class DailyBlogs(BasicNewsRecipe):
    title                 = u'Daily Blog Roll'
    description           = "Collection of Daily Blog Reading"
    oldest_article        = 1
    max_articles_per_feed = 5
    added_links           = []
    remove_tags           = [
        dict(name='a',   attrs={
            'class':'a2a_dd addtoany_share_save'}),

        dict(name='div', attrs={
            'class':['Introduction','divider']}),

        dict(name='div', attrs={
            'id':['feature', 'podcast']}),

        dict(name='div', attrs={
            'id':re.compile(r'follow.*', re.DOTALL|re.IGNORECASE)}),

        dict(name='hr'),
                             ]

    language              = 'en'
    remove_empty_feeds    = True
    remove_javascript     = True
    no_stylesheets        = True

    extra_css = """
h1 {
  font-family:Arial,Helvetica,sans-serif;
  font-weight:bold;
  font-size:large;
}

h2 {
  font-family:Arial,Helvetica,sans-serif;
  font-weight:normal;
  font-size: 2.6em;
}

p {
  font-family:Arial,Helvetica,sans-serif;
  font-size:small;
}

body {
  font-family:Helvetica,Arial,sans-serif;
  font-size:small;
}
"""

    feeds = [
(u'Weird Things',    u'feed://worldofweirdthings.com/feed/'),
(u'Bad Astronomy',   u'feed://feeds.feedburner.com/BadAstronomyBlog?format=xml'),
(u'Skepticblog',     u'feed://skepticblog.org/feed/'),
(u'The Skeptic',     u'http://www.skeptic.com/feed'),
(u'E-Skeptic',       u'http://www.skeptic.com/eskeptic'),
(u'All-SkepticBlog', u'http://skepticblog.org/feed'),
(u'Brian Dunning',   u'http://skepticblog.org/author/dunning/feed/'),
(u'Daniel Loxton',   u'http://skepticblog.org/author/loxton/feed/'),
(u'Kirsten Sanford', u'http://skepticblog.org/author/sanford/feed/'),
(u'Mark Edward',     u'http://skepticblog.org/author/edward/feed/'),
(u'Michael Shermer', u'http://skepticblog.org/author/shermer/feed/'),
(u'Phil Plait',      u'http://skepticblog.org/author/plait/feed/'),
(u'Ryan Johnson',    u'http://skepticblog.org/author/johnson/feed/'),
(u'Steven Novella',  u'http://skepticblog.org/author/novella/feed/'),
(u'Yau-Man Chan',    u'http://skepticblog.org/author/chan/feed/'),
        ]

    cover_url = "http://skepticblog.org/wp-content/uploads/Fairy_making_of_15_S.jpg"

    def get_browser(self):
      br = BasicNewsRecipe.get_browser(self)
      br.addheaders = [('Accept', 'text/html')]
      return br

[Starson17]

Originally Posted by krunk

The documentation says extra_css should be placed in the <style></style> tags of the <head>. So I checked in output/index.html to be sure and nothing was placed their either.

AFAIK, it appears in stylesheet.css.

Originally Posted by krunk

The modified recipe looks like:

    extra_css = """
h1 {
  font-family:Arial,Helvetica,sans-serif;
  font-weight:bold;
  font-size:large;
}
"""

You need indents in the extra_css, just like elsewhere, or they'll get ignored.

[Starson17]

Originally Posted by Thetasquared

I would like to request a recipe for the "Current Issue" of Science News.

The rss feed is:
http://www.sciencenews.org/view/feed...ame/issues.rss

I know that a Science News recipe exists, but simply switching the feeds does not return articles for the "current issue"

Here's a quick and dirty version. Why don't you look it over and spot what needs to get cleaned up better. Post here and I'll address it. I really like Science News.

Spoiler:

#!/usr/bin/env  python

__license__   = 'GPL v3'
'''
sciencenews.org
'''
from calibre.web.feeds.news import BasicNewsRecipe

class Sciencenews(BasicNewsRecipe):
    title                 = u'Science News Current Issues'
    __author__            = u'Starson17'
    description           = u"Science News is an award-winning weekly newsmagazine covering the most important research in all fields of science. Its 16 pages each week are packed with short, accurate articles that appeal to both general readers and scientists. Published since 1922, the magazine now reaches about 150,000 subscribers and more than 1 million readers. These are the latest News Items from Science News."
    oldest_article        = 30
    language = 'en'

    max_articles_per_feed = 100
    no_stylesheets        = True
    use_embedded_content  = False
    timefmt               = ' [%A, %d %B, %Y]'
    recursions = 1
    
    extra_css = '''
                .content_description{font-family:georgia ;font-size:x-large; color:#646464 ; font-weight:bold;}
                .content_summary{font-family:georgia ;font-size:small ;color:#585858 ; font-weight:bold;}
                .content_authors{font-family:helvetica,arial ;font-size: xx-small ;color:#14487E ;}
                .content_edition{font-family:helvetica,arial ;font-size: xx-small ;}
                .exclusive{color:#FF0000 ;}
                .anonymous{color:#14487E ;}
                .content_content{font-family:helvetica,arial ;font-size: x-small ; color:#000000;}
                .description{color:#585858;font-family:helvetica,arial ;font-size: xx-small ;}
                .credit{color:#A6A6A6;font-family:helvetica,arial ;font-size: xx-small ;}
                '''

    keep_only_tags = [ dict(name='div', attrs={'id':'column_action'}) ]
    remove_tags_after = dict(name='ul', attrs={'id':'content_functions_bottom'})
    remove_tags = [
                     dict(name='ul', attrs={'id':'content_functions_bottom'})
                    ,dict(name='div', attrs={'id':['content_functions_top','breadcrumb_content']})
                    ,dict(name='img', attrs={'class':'icon'})
                    ,dict(name='div', attrs={'class': 'embiggen'})
                  ]
    
    feeds       = [(u"Science News Current Issues", u'http://www.sciencenews.org/view/feed/type/edition/name/issues.rss')]

    match_regexps = [
            r'www.sciencenews.org/view/feature/id/',
            r'www.sciencenews.org/view/generic/id'
            ]
    #http://www.sciencenews.org/view/feature/id/63177/title/Fire_%2Bamp%3B_Ice
    
    def get_cover_url(self):
        cover_url = None
        index = 'http://www.sciencenews.org/view/home'
        soup = self.index_to_soup(index)
        link_item = soup.find(name = 'img',alt = "issue")
        print link_item
        if link_item:
           cover_url = 'http://www.sciencenews.org' + link_item['src'] + '.jpg'

        return cover_url

    def preprocess_html(self, soup):
        for tag in soup.findAll(name=['span']):
            tag.name = 'div'
        return soup

[Thetasquared]

Originally Posted by Starson17

That is not a feed for the "current issue", and it doesn't point to articles of the current issue. It's a feed for multiple issues and each feed, instead of pointing to a typical article, points to a web page with links to the articles in that issue.

Are you looking for one issue and its articles, or all the issues in that feed and all the articles in them?

It looks interesting, so I may tackle it.

wow! yes you're right LOL... Ok, so what I would like is to download all of the articles in the most recent issue of that feed. Each time a new issue comes out I can dl it to Calibre.

Hope you take on the task, and thank you for your interest!


- Teddy

[Starson17]

Originally Posted by Thetasquared

wow! yes you're right LOL... Ok, so what I would like is to download all of the articles in the most recent issue of that feed. Each time a new issue comes out I can dl it to Calibre.

Hope you take on the task, and thank you for your interest!
- Teddy

A first draft is already posted - the ball's in your court now. (It's currently multiple issues, not single issue, but that's not hard to fix later, once the format is cleaned up.)

[Thetasquared]

Ha! yeah so in the time it took me to respond to you you've already made a first draft.

attempting to download right now. Seems to be working.

[Thetasquared]

Originally Posted by Starson17

A first draft is already posted - the ball's in your court now. (It's currently multiple issues, not single issue, but that's not hard to fix later, once the format is cleaned up.)

It worked! downloaded the issue and it looks fine in Calibre (not near my ereader right now). I know next to nothing about coding, so I don't know what to clean up. Only thing I noticed is that the words "Fire & Ice" are in the code. Does that mean if the current issue changes it will still download the "fire & ice" issue?

I love Science News too! Thank you for your time and effort in making this recipe work. It will add tons of enjoyment to my life! umm.. and knowledge. will increase my knowledge of science stuff :-)