The Economist recipe does not work

[gc886]

I have been using the recipe for The Economist for years and never had any problem. Today it seem to have failed due to a TypeError. I am running version 7.12.0, not the latest, and I am not sure if I need to update to the latest Calibre version to make this work. Is anyone else also experiencing this and is there any workaround?

Error details copied here:

calibre, version 7.12.0 (linux, embedded-python: False)
Conversion error: Failed: Fetch news from The Economist

Fetch news from The Economist
Conversion options changed from defaults:
  verbose: 2
Resolved conversion options
calibre version: 7.12.0
{'add_alt_text_to_img': False,
 'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x7b789b82f7a0>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_file_type': 'old',
 'mobi_ignore_margins': False,
 'mobi_keep_original_images': False,
 'mobi_toc_at_start': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': False,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.OutputProfile object at 0x7b789b7e9a00>,
 'page_breaks_before': None,
 'personal_doc': '[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'share_not_sync': False,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'transform_html_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:economist
Trying to get latest version of recipe: economist
Using user agent: common_words/based
Traceback (most recent call last):
  File "/usr/bin/calibre-parallel", line 21, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/calibre/calibre/utils/ipc/worker.py", line 215, in main
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/gui2/convert/gui_conversion.py", line 31, in gui_convert_recipe
    gui_convert('from-gui.recipe', output, recommendations, notification=notification,
  File "/usr/lib/calibre/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
    plumber.run()
  File "/usr/lib/calibre/calibre/ebooks/conversion/plumber.py", line 1127, in run
    self.oeb = self.input_plugin(stream, self.opts,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/customize/conversion.py", line 245, in __call__
    ret = self.convert(stream, options, file_ext,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/ebooks/conversion/plugins/recipe_input.py", line 138, in convert
    ro.download()
  File "/usr/lib/calibre/calibre/web/feeds/news.py", line 1069, in download
    res = self.build_index()
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/web/feeds/news.py", line 1248, in build_index
    feeds = feeds_from_index(self.parse_index(), oldest_article=self.oldest_article,
                             ^^^^^^^^^^^^^^^^^^
  File "<string>", line 249, in parse_index
TypeError: can only concatenate str (not "dict") to str

[kovidgoyal]

Update calibre and you will be fine or wait an hour and retry and you will be fine.

[gc886]

Thank you! The recipe works now!

[carol.land-0y]

I am still having issues with the Economist download. It seems to error out at the end after downloading a significant percentage of the file.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre/web/fetch/simple.py", line 434, in process_images
  File "calibre/web/fetch/simple.py", line 292, in fetch_url
calibre.web.fetch.simple.FetchError: Too Many Requests

Fetching https://www.economist.com/cdn-cgi/image/width=600,quality=80,format=auto/media-assets/image/20240810_INT401.png
Fetched https://www.economist.com/cdn-cgi/image/width=600,quality=80,format=auto/media-assets/image/20240810_INT401.png in 1.076916 seconds
Recursion limit reached. Skipping links in file:////tmp/calibre_7.16.0_tmp_3zsv8jp1/i614yys9.html
file:////tmp/calibre_7.16.0_tmp_3zsv8jp1/i614yys9.html saved to /tmp/calibre_7.16.0_tmp_3zsv8jp1/7hud9dhs_plumber/feed_18/article_0/i614yys9.xhtml
Fetching file:////tmp/calibre_7.16.0_tmp_3zsv8jp1/9namj3e4.html
Fetched //tmp/calibre_7.16.0_tmp_3zsv8jp1/9namj3e4.html in 0.0 seconds
Downloaded article: Economic data, commodities and markets from https://www.economist.com/economic-and-financial-indicators/2024/08/08/economic-data-commodities-and-markets
Parsed file:////tmp/calibre_7.16.0_tmp_3zsv8jp1/9namj3e4.html in 0.0 seconds
Processing images...
Fetching https://www.economist.com/cdn-cgi/image/width=600,quality=80,format=auto/media-assets/image/20240810_OBP001.jpg
Could not fetch image  https://www.economist.com/cdn-cgi/image/width=600,quality=80,format=auto/media-assets/image/20240810_OBP001.jpg
Traceback (most recent call last):
  File "calibre/web/fetch/simple.py", line 287, in fetch_url
  File "mechanize/_mechanize.py", line 241, in open_novisit
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 429: Too Many Requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre/web/fetch/simple.py", line 434, in process_images
  File "calibre/web/fetch/simple.py", line 292, in fetch_url
calibre.web.fetch.simple.FetchError: Too Many Requests

Recursion limit reached. Skipping links in file:////tmp/calibre_7.16.0_tmp_3zsv8jp1/9namj3e4.html
file:////tmp/calibre_7.16.0_tmp_3zsv8jp1/9namj3e4.html saved to /tmp/calibre_7.16.0_tmp_3zsv8jp1/7hud9dhs_plumber/feed_19/article_0/9namj3e4.xhtml
Downloaded article: Inna Solovyova studied both stagecraft and the Russian soul from https://www.economist.com/obituary/2024/08/08/inna-solovyova-studied-both-stagecraft-and-the-russian-soul
Traceback (most recent call last):
  File "runpy.py", line 198, in _run_module_as_main
  File "runpy.py", line 88, in _run_code
  File "site.py", line 47, in <module>
  File "site.py", line 43, in main
  File "calibre/utils/ipc/worker.py", line 215, in main
  File "calibre/gui2/convert/gui_conversion.py", line 31, in gui_convert_recipe
  File "calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
  File "calibre/ebooks/conversion/plumber.py", line 1129, in run
  File "calibre/customize/conversion.py", line 245, in __call__
  File "calibre/ebooks/conversion/plugins/recipe_input.py", line 155, in convert
  File "calibre/web/feeds/news.py", line 1099, in download
  File "calibre/web/feeds/news.py", line 1370, in build_index
  File "calibre/web/feeds/news.py", line 1522, in create_opf
  File "<string>", line 220, in publication_date
  File "mechanize/_mechanize.py", line 257, in open
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 429: Too Many Requests

[kovidgoyal]

That error just means you will have some missing images. It looks like for some reason the economist servers are throttling you, I just tried downloading the economist and it worked fine with no errors for me. Are you sure you are using the builtin recipe?

[BillD]

Same error last couple of days - http error 429 "too many requests" - maybe Economist has changed something. Will try VPN or something to see if that matters.

edit: nevermind - switched to a VPN - download worked fine

[unkn0wn]

its the images that fail to fetch, and its pulling down the whole thing.

there's already a 1 sec delay, increase the delay = 2 and see if it works.
here: https://github.com/kovidgoyal/calibr...st.recipe#L198

do tell if it works, i'll make those changes.

[makmende]

In the new Economist recipe, how do I download older editions?

[unkn0wn]

in fetch news dialog, use advanced menu, enter the edition date in the date field.
if using CLI, use recipe_specific_options, but only the last week or two are possible with the new recipe.

[yuefyukey]

I have a question.

Before Amazon Kindle was switched to EPUB from MOBI i could easily navigate in the menus and articles in Economist issues.
Now since its forced to EPUB, whole interface looks very primitive and cant see articles in menu (only sections)

Currently using Amazon Kindle Paperwhite 2

I miss the old MOBI version.

[kovidgoyal]

The kindle does not support EPUB only Amazons send to device service auto converts EPUB before sending the file to the kindle. Plug in your Kindle using a USB connectiona nd continue to send news to it in MOBI format with calibre and you will have all the peridical specific navigation.

[yuefyukey]

Thanks for quick response. I connected via usb, even though calibre says economist is sent to device it does not appear on my kindle.

I am using macbook pro m1. Does it make a difference?

[kovidgoyal]

No it doesnt make any difference.