Is it just me or are others experiencing this too?

[TonytheBookworm]

Alright, for the life of me I don't know what the deal is with my "test environment". I reported this as a potential bug but Kovid stated that it wasn't that I just needed to add more remove_tags and such. Well Yeah
The issue at hand is still this. If i take and run ebook-convert test1.recipe out_put_dir --test -vv > myrecipe.txt

Yes, I know it only gets too feeds. but the problem is in "most not all but most cases" it cleans the dang junk up. So when i run the test after using remove_tags then I see no junk I automatically assume that it is fine. But that is simply not the case. For instance the popular science code that I submitted for 7.20 I was led to believe it worked fine because the console test looked perfect. yet when i look run the built in recipe in 7.20 i get junk. I ran the recipe without --test and the same exact article(s) that showed perfect when using the --test switch show junk.

So what is the suggestion? To not use --test and run full feed parsing? The test just makes it faster to well "test"
here is the code if your interesting in seeing what I mean. I mean yeah i realize that it needs more remove_tags and keep_only tags but that is not the issue. The issue is why does the same article from the same feed with the same code show two totally different results when i use --test vs not using --test ?

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup, re

class AdvancedUserRecipe1282101454(BasicNewsRecipe):
    title = 'Popular Science'
    language = 'en'
    __author__ = 'TonytheBookworm'
    description = 'Popular Science'
    publisher = 'Popular Science'
    category = 'gadgets,science'
    oldest_article = 7 # change this if you want more current articles. I like to go a week in
    max_articles_per_feed = 100
    no_stylesheets = True
    remove_javascript = True
    
    masthead_url = 'http://www.raytheon.com/newsroom/rtnwcm/groups/Public/documents/masthead/rtn08_popscidec_masthead.jpg'
    
    remove_tags = [dict(name='div', attrs={'id':['toolbar','main_supplements']}),
                   dict(name='span', attrs={'class':['comments']}),
                   dict(name='div', attrs={'class':['relatedinfo related-right','node_navigation','content2']}),
                   dict(name='ul', attrs={'class':['item-list clear-block']})]                     
    feeds          = [
                      
                      ('Gadgets', 'http://www.popsci.com/full-feed/gadgets'),
                      ('Cars', 'http://www.popsci.com/full-feed/cars'),
                      ('Science', 'http://www.popsci.com/full-feed/science'),
                      ('Technology', 'http://www.popsci.com/full-feed/technology'),
                      ('DIY', 'http://www.popsci.com/full-feed/diy'),
                      
                    ]

    
 #The following will get read of the Gallery: links when found    
        
    def preprocess_html(self, soup) :
        print 'SOUP IS: ', soup
        weblinks = soup.findAll(['head','h2'])
        if weblinks is not None:
            for link in weblinks:
                if re.search('(Gallery)(:)',str(link)):
                  
                  link.parent.extract()
        return soup
  #-----------------------------------------------------------------

Kovid: I will submit an update to you soon for Popular Science as well. Seems like both of my submissions for 7.20 were no go for launch That's what i get for assuming all was fine when using the --test switch

[TonytheBookworm]

Someone please test this code on your end and see if you get any junk. I don't want to keep submitting what I believe to be working code to Kovid and then turning around looking like a moron when it ends up looking like crap

Thanks....

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup, re

class AdvancedUserRecipe1282101454(BasicNewsRecipe):
    title = 'Popular Science'
    language = 'en'
    __author__ = 'TonytheBookworm'
    description = 'Popular Science'
    publisher = 'Popular Science'
    category = 'gadgets,science'
    oldest_article = 7 # change this if you want more current articles. I like to go a week in
    max_articles_per_feed = 100
    no_stylesheets = True
    remove_javascript = True
    use_embedded_content = True
    
    masthead_url = 'http://www.raytheon.com/newsroom/rtnwcm/groups/Public/documents/masthead/rtn08_popscidec_masthead.jpg'
    
               
    feeds          = [
                      
                      ('Gadgets', 'http://www.popsci.com/full-feed/gadgets'),
                      ('Cars', 'http://www.popsci.com/full-feed/cars'),
                      ('Science', 'http://www.popsci.com/full-feed/science'),
                      ('Technology', 'http://www.popsci.com/full-feed/technology'),
                      ('DIY', 'http://www.popsci.com/full-feed/diy'),
                      
                    ]

    
 #The following will get read of the Gallery: links when found    
        
    def preprocess_html(self, soup) :
        print 'SOUP IS: ', soup
        weblinks = soup.findAll(['head','h2'])
        if weblinks is not None:
            for link in weblinks:
                if re.search('(Gallery)(:)',str(link)):
                  
                  link.parent.extract()
        return soup
  #-----------------------------------------------------------------

***Starson17 - I used the use_embedded_content flag that i didn't know anything about until you mentioned it. Makes some feeds a looooot easier. Thanks

[Starson17]

Originally Posted by TonytheBookworm

***Starson17 - I used the use_embedded_content flag that i didn't know anything about until you mentioned it. Makes some feeds a looooot easier. Thanks

I'm glad it helped.

[Starson17]

Originally Posted by TonytheBookworm

Someone please test this code on your end and see if you get any junk.

No junk both ways w/ and w/o --test. You'll have to show me exactly what you're seeing for me to look more closely. You're a big boy now, too - track it down - set up to run from code.
Look at lines 595-597 of news.py. Search for "test" in that file.

[TonytheBookworm]

Here are some snap shots that should explain what I'm seeing. As far as the news.py I tried to find that file without any look. The closes I came by looking in my myrecipe.txt was a file that was suppose to be in
\site-packages\calibre\web\feeds
I went there and the only file that I seen in there was news.py0 I tried to open that file with ultra edit and it is binary..
thanks.

[Starson17]

Originally Posted by TonytheBookworm

Here are some snap shots that should explain what I'm seeing.

I'm not seeing any diff between --test and not.

As far as the news.py I tried to find that file without any look.

You have to run from source.
http://calibre-ebook.com/user_manual/develop.html

[DoctorOhh]

Originally Posted by TonytheBookworm

Someone please test this code on your end and see if you get any junk. I don't want to keep submitting what I believe to be working code to Kovid and then turning around looking like a moron when it ends up looking like crap

I tested this code by loading the recipe in the GUI and running it. I don't see any of the junk you are seeing. It seems to be a nice clean output.