The Economist recipe does not work

[gc886]

I have been using the recipe for The Economist for years and never had any problem. Today it seem to have failed due to a TypeError. I am running version 7.12.0, not the latest, and I am not sure if I need to update to the latest Calibre version to make this work. Is anyone else also experiencing this and is there any workaround?

Error details copied here:

calibre, version 7.12.0 (linux, embedded-python: False)
Conversion error: Failed: Fetch news from The Economist

Fetch news from The Economist
Conversion options changed from defaults:
  verbose: 2
Resolved conversion options
calibre version: 7.12.0
{'add_alt_text_to_img': False,
 'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x7b789b82f7a0>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_file_type': 'old',
 'mobi_ignore_margins': False,
 'mobi_keep_original_images': False,
 'mobi_toc_at_start': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': False,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.OutputProfile object at 0x7b789b7e9a00>,
 'page_breaks_before': None,
 'personal_doc': '[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'share_not_sync': False,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'transform_html_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:economist
Trying to get latest version of recipe: economist
Using user agent: common_words/based
Traceback (most recent call last):
  File "/usr/bin/calibre-parallel", line 21, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/calibre/calibre/utils/ipc/worker.py", line 215, in main
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/gui2/convert/gui_conversion.py", line 31, in gui_convert_recipe
    gui_convert('from-gui.recipe', output, recommendations, notification=notification,
  File "/usr/lib/calibre/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
    plumber.run()
  File "/usr/lib/calibre/calibre/ebooks/conversion/plumber.py", line 1127, in run
    self.oeb = self.input_plugin(stream, self.opts,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/customize/conversion.py", line 245, in __call__
    ret = self.convert(stream, options, file_ext,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/ebooks/conversion/plugins/recipe_input.py", line 138, in convert
    ro.download()
  File "/usr/lib/calibre/calibre/web/feeds/news.py", line 1069, in download
    res = self.build_index()
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/web/feeds/news.py", line 1248, in build_index
    feeds = feeds_from_index(self.parse_index(), oldest_article=self.oldest_article,
                             ^^^^^^^^^^^^^^^^^^
  File "<string>", line 249, in parse_index
TypeError: can only concatenate str (not "dict") to str

[kovidgoyal]

Update calibre and you will be fine or wait an hour and retry and you will be fine.

[gc886]

Thank you! The recipe works now!

[carol.land-0y]

I am still having issues with the Economist download. It seems to error out at the end after downloading a significant percentage of the file.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre/web/fetch/simple.py", line 434, in process_images
  File "calibre/web/fetch/simple.py", line 292, in fetch_url
calibre.web.fetch.simple.FetchError: Too Many Requests

Fetching https://www.economist.com/cdn-cgi/image/width=600,quality=80,format=auto/media-assets/image/20240810_INT401.png
Fetched https://www.economist.com/cdn-cgi/image/width=600,quality=80,format=auto/media-assets/image/20240810_INT401.png in 1.076916 seconds
Recursion limit reached. Skipping links in file:////tmp/calibre_7.16.0_tmp_3zsv8jp1/i614yys9.html
file:////tmp/calibre_7.16.0_tmp_3zsv8jp1/i614yys9.html saved to /tmp/calibre_7.16.0_tmp_3zsv8jp1/7hud9dhs_plumber/feed_18/article_0/i614yys9.xhtml
Fetching file:////tmp/calibre_7.16.0_tmp_3zsv8jp1/9namj3e4.html
Fetched //tmp/calibre_7.16.0_tmp_3zsv8jp1/9namj3e4.html in 0.0 seconds
Downloaded article: Economic data, commodities and markets from https://www.economist.com/economic-and-financial-indicators/2024/08/08/economic-data-commodities-and-markets
Parsed file:////tmp/calibre_7.16.0_tmp_3zsv8jp1/9namj3e4.html in 0.0 seconds
Processing images...
Fetching https://www.economist.com/cdn-cgi/image/width=600,quality=80,format=auto/media-assets/image/20240810_OBP001.jpg
Could not fetch image  https://www.economist.com/cdn-cgi/image/width=600,quality=80,format=auto/media-assets/image/20240810_OBP001.jpg
Traceback (most recent call last):
  File "calibre/web/fetch/simple.py", line 287, in fetch_url
  File "mechanize/_mechanize.py", line 241, in open_novisit
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 429: Too Many Requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre/web/fetch/simple.py", line 434, in process_images
  File "calibre/web/fetch/simple.py", line 292, in fetch_url
calibre.web.fetch.simple.FetchError: Too Many Requests

Recursion limit reached. Skipping links in file:////tmp/calibre_7.16.0_tmp_3zsv8jp1/9namj3e4.html
file:////tmp/calibre_7.16.0_tmp_3zsv8jp1/9namj3e4.html saved to /tmp/calibre_7.16.0_tmp_3zsv8jp1/7hud9dhs_plumber/feed_19/article_0/9namj3e4.xhtml
Downloaded article: Inna Solovyova studied both stagecraft and the Russian soul from https://www.economist.com/obituary/2024/08/08/inna-solovyova-studied-both-stagecraft-and-the-russian-soul
Traceback (most recent call last):
  File "runpy.py", line 198, in _run_module_as_main
  File "runpy.py", line 88, in _run_code
  File "site.py", line 47, in <module>
  File "site.py", line 43, in main
  File "calibre/utils/ipc/worker.py", line 215, in main
  File "calibre/gui2/convert/gui_conversion.py", line 31, in gui_convert_recipe
  File "calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
  File "calibre/ebooks/conversion/plumber.py", line 1129, in run
  File "calibre/customize/conversion.py", line 245, in __call__
  File "calibre/ebooks/conversion/plugins/recipe_input.py", line 155, in convert
  File "calibre/web/feeds/news.py", line 1099, in download
  File "calibre/web/feeds/news.py", line 1370, in build_index
  File "calibre/web/feeds/news.py", line 1522, in create_opf
  File "<string>", line 220, in publication_date
  File "mechanize/_mechanize.py", line 257, in open
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 429: Too Many Requests

[kovidgoyal]

That error just means you will have some missing images. It looks like for some reason the economist servers are throttling you, I just tried downloading the economist and it worked fine with no errors for me. Are you sure you are using the builtin recipe?

[BillD]

Same error last couple of days - http error 429 "too many requests" - maybe Economist has changed something. Will try VPN or something to see if that matters.

edit: nevermind - switched to a VPN - download worked fine

[unkn0wn]

its the images that fail to fetch, and its pulling down the whole thing.

there's already a 1 sec delay, increase the delay = 2 and see if it works.
here: https://github.com/kovidgoyal/calibr...st.recipe#L198

do tell if it works, i'll make those changes.

[makmende]

In the new Economist recipe, how do I download older editions?

[unkn0wn]

in fetch news dialog, use advanced menu, enter the edition date in the date field.
if using CLI, use recipe_specific_options, but only the last week or two are possible with the new recipe.