Foreign Affairs fails to fetch

[tamur93]

Hello, i have a foreign affairs subscription. The first one i dowloaded just fine, however i have tried again after the new magazine came out and it gave me this error message.

calibre, version 2.27.0 (darwin, isfrozen: True)
Conversion Error: Failed: Fetch news from Foreign Affairs (Subcription)

Fetch news from Foreign Affairs (Subcription)
Resolved conversion options
calibre version: 2.27.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x10b1d03d0>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_file_type': 'old',
 'mobi_ignore_margins': False,
 'mobi_keep_original_images': False,
 'mobi_toc_at_start': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': True,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindlePaperWhiteOutput object at 0x10b1d0b10>,
 'page_breaks_before': None,
 'personal_doc': '[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'share_not_sync': False,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
Python function terminated unexpectedly: no control matching name 'name'
InputFormatPlugin: Recipe Input running
Using custom recipe
Traceback (most recent call last):
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 209, in main
    return run_entry_point()
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 114, in run_entry_point
    return getattr(pmod, func)()
  File "site-packages/calibre/utils/ipc/worker.py", line 193, in main
  File "site-packages/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
  File "site-packages/calibre/ebooks/conversion/plumber.py", line 1042, in run
  File "site-packages/calibre/customize/conversion.py", line 241, in __call__
  File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 116, in convert
  File "site-packages/calibre/web/feeds/news.py", line 918, in __init__
  File "<string>", line 123, in get_browser
  File "site-packages/mechanize/_form.py", line 2780, in __setitem__
  File "site-packages/mechanize/_form.py", line 3101, in find_control
  File "site-packages/mechanize/_form.py", line 3185, in _find_control
mechanize._form.ControlNotFoundError: no control matching name 'name'

What does this mean? How can i fix this error? Any help would be appreciated. Thanks !

[PeterT]

It looks as if maybe the format of the magazine has changed, and the recipe was not able to logon to the website

mechanize._form.ControlNotFoundError: no control matching name 'name'

The recipe probably needs to be updated.

[anisotrope]

Hi, I think I fixed the recipe. I left in some debug strings. ymmv:

from calibre.web.feeds.news import BasicNewsRecipe
import re


def select_form(form):
    return form.attrs.get('class', None) == 'user-login-form'

    
class ForeignAffairsRecipe(BasicNewsRecipe):
    ''' there are three modifications:
    1) fetch issue cover
    2) toggle ignore premium articles
    3) extract proper section names, ie. "Comments", "Essay"

    by Chen Wei, 2012-02-05
	
	Additional modifications to support rebranded website
	
	by anisotrope, 27 June 2015
	'''

    __license__  = 'GPL v3'
    __author__ = 'Rick Shang, kwetal, anisotrope'
    language = 'en'
    version = 1.02

    title = u'Foreign Affairs (Subcription)'
    publisher = u'Council on Foreign Relations'
    category = u'USA, Foreign Affairs'
    description = u'The leading forum for serious discussion of American foreign policy and international affairs.'

    no_stylesheets = True
    remove_javascript = True
    needs_subscription = True

    INDEX = 'http://www.foreignaffairs.com'
    FRONTPAGE = 'http://www.foreignaffairs.com/magazine'


    remove_tags = [dict(name = 'svg')]
    remove_tags_before = dict(name = 'div', attrs = {'class': 'print-content'})
	#remove_tags_before =[
	#						dict(name = 'div', attrs = {'class': 'print-content'}),
	#						dict(name = 'h1', attrs = {'class': 'article-header__headline'}),
	#					]
    remove_tags_after = dict(name = 'div', attrs = {'class': 'print-footer'})

    extra_css = '''
                body{font-family:verdana,arial,helvetica,geneva,sans-serif;}
                div.print-footer {font-size: x-small; color: #696969;}
                '''

    conversion_options = {'comments': description, 'tags': category, 'language': 'en',
                          'publisher': publisher}

    temp_files = []

    def get_cover_url(self):
        soup = self.index_to_soup(self.FRONTPAGE)
        div = soup.find('div', attrs={'class':'magazine-hero__image image_auto_width'})
        img_url =  div.find('img')['src']
        return img_url #The url includes the https:// as necessary

    def get_print_url(self, url):
        print "Checking url consistency: {}".format(url is url.strip())
        print "Checking url: {}".format(url.strip())
        article_soup = self.index_to_soup(url.strip())
        print "Checking none: {}".format(article_soup is not None)
        print "Checking type: {}".format(type(article_soup))

        if article_soup is not None:
            shortlink = article_soup.find('a', attrs={'class':re.compile(r'\bicon-print\b')})
            if shortlink:
                print "Found shortlink through a/class"
                print shortlink
                print "Article print-version url: {}".format(shortlink['href'])
                return shortlink['href']
            else: 
                return url
        else :
            return url

        
    def parse_index(self):

        answer = []
        soup = self.index_to_soup(self.FRONTPAGE)
        #get dates
        date = re.split('\s\|\s',self.tag_to_string(soup.head.title.string))[0]
        self.title = "Foreign Affairs ({})".format(date)
        self.timefmt =  u' [%s]'%date

        sec_start = soup.findAll('section', attrs= {'class':re.compile(r'\bmagazine-list\b')})
        for sec in sec_start:
            articles = []
            section = self.tag_to_string(sec.find('h1'))
            print "Section: " + section
            for article_block in sec.findAll('article'):
                if article_block.find('a') is not None:
                    title=self.tag_to_string(article_block.div.a.h2)
                    article_url = article_block.div.a['href']
                    print "Article url: {}".format(article_url)
                    url = self.get_print_url(article_url)
                    atr=article_block.findNext('p', attrs = {'class': 'author'})
                    if atr is not None:
                        author=self.tag_to_string(atr)
                    else:
                        author=''
                    desc=article_block.findNext('div', attrs = {'class': 'deck'})
                    if desc is not None:
                        description=self.tag_to_string(desc)
                    else:
                        description=''
                    articles.append({'title':title, 'date':None, 'url':url, 'description':description, 'author':author})
            if articles:
                answer.append((section, articles))
        return answer

    def preprocess_html(self, soup):
        for img in soup.findAll('img', attrs = {'src': True}):
            if not img['src'].startswith('http'):
                img['src'] = self.INDEX + img['src']

        return soup

    def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
        if self.username is not None and self.password is not None:
            page = br.open('https://www.foreignaffairs.com/user?destination=user%3Fop%3Dlo')
            br.select_form( predicate = select_form )
            br.form['name'] = self.username
            br.form['pass'] = self.password
            br.submit()
        return br

    def cleanup(self):
        self.browser.open('https://www.foreignaffairs.com/user/logout')

[tamur93]

TY so much, worked like a charm!

[mendesitba]

Hello guys, I just downloaded the last foreign affairs issue and I have to say, this recipe is awesome!!!

I like it so much that now I want to download every issue since the last 2 years lol. Anyone knows if there is anyway to do it with this recipe? Is there something in the recipe we can change to download past issues?

PS: I didn't want to create another thread, so I asked it here. If I can't get an answer through this thread, I'll create another one.

[anisotrope]

Originally Posted by mendesitba

I like it so much that now I want to download every issue since the last 2 years lol. Anyone knows if there is anyway to do it with this recipe? Is there something in the recipe we can change to download past issues?

So I think if you change the following line in the recipe:

FRONTPAGE = 'http://www.foreignaffairs.com/magazine'

To the URL of the issue you want to download, for example: "https://www.foreignaffairs.com/issues/2015/94/3", it should work. I think the regular url ("http://www.foreignaffairs.com/magazine") just forwards to the specific issue anyway, so there's not much of a difference. Let us know if that works, I was planning on trying that myself at some point.

[mendesitba]

Originally Posted by anisotrope

So I think if you change the following line in the recipe:

FRONTPAGE = 'http://www.foreignaffairs.com/magazine'

To the URL of the issue you want to download, for example: "https://www.foreignaffairs.com/issues/2015/94/3", it should work. I think the regular url ("http://www.foreignaffairs.com/magazine") just forwards to the specific issue anyway, so there's not much of a difference. Let us know if that works, I was planning on trying that myself at some point.

YESSSSS!!! It worked perfectly!!!

It was so simple, but I don't have any experience at all in programming, so I really don't know what these things like "INDEX" or "FRONTPAGE" means.


PS: To read the older issues of the Foreign Affairs I used to download the PDF version on the site, crop the page numbers and remove each ads and each image with Adobe, transform the outcome to HTML with Mobipocket Creator, and finally use the HTML to create a MOBI version of the Magazine with Calibre. And even then the result was far from perfect. Now, I don't have to do all this anymore!!!