Guardian/Observer recipe broken again!

[paddyrm]

Sorry Kovid, they seem to have made some changes again, and only the cover/front page is scraped for five days now.
Regards
Paddy

[paddyrm]

Kovid, have managed to solve the problem myself, by amending your script for the Observer on Sundays. It looks like they changed the weekday paper to the Sunday format. So stand easy!
Cheers
Paddy

[kovidgoyal]

That's great but post your updated recipe so others can benefit.

[dre]

Same problem with The Guardian. I get only 2 subjects: Unknown feed with 10 items and Sport. I hope a solution will come. I'm a member of The Guardian but reading it in Calibre is my favorite way. Greetings

[paddyrm]

On Kovid's suggestion (blame him!) I have uploaded the two recipes that work at the moment, one for the daily Guardian and one for the Sunday Observer.
Paddy

[kovidgoyal]

https://github.com/kovidgoyal/calibr...57cedc5739c7f9

[dre]

I've a problem in making a new feed with the github file.

I expanded the code at several places and copied and pasted it into a new feed. When clicking the OK button i got a failed compile message.

FOUT: Ongeldig recept: Failed to compile the recipe, with syntax error: expected an indented block after function definition on line 96 (<string>, line 97)

See attached file with copy of selection in github file 'String 96 github file'
and file with copy of selection after copy-paste github file in create additional newsfeed String 96 after copy pase'

Did I oversee something?

[dre]

I again downloaded the unchanged newsfeed recipe [guardian observer] in Calibre and stunning surprise, the feed was OK as usual.
The only thing I did to this recipe was pasting the new code from github and after the 'Failed to compile the recipe' message [syntax error on a previous block] I didn't save it. So in fact I changed nothing.
I don't understand but happy to read the news!
Thanks

[kovidgoyal]

You dont need to do anything builtin recipes are auto updated.

[dre]

In that case, thanks very much

[Xer00]

Hi, i’m getting a ‘conversion’ error for the last month or so which is not letting me download The Guardian. Is this a problem with my calibre settings? I’m not sure i changed anything since it stopped working:

calibre, version 7.11.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from The Guardian and The Observer

Fetch news from The Guardian and The Observer
Conversion options changed from defaults:
output_profile: 'kobo'
verbose: 2
Resolved conversion options
calibre version: 7.11.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '3',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x0000013A3AA5F610>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'kepub_clean_markup': True,
'kepub_disable_hyphenation': False,
'kepub_hyphenate': True,
'kepub_hyphenate_chars': 6,
'kepub_hyphenate_chars_after': 3,
'kepub_hyphenate_chars_before': 3,
'kepub_hyphenate_limit_lines': 2,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.KoboReaderOutput object at 0x0000013A3AA77190>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:guardian
Trying to get latest version of recipe: guardian
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
Traceback (most recent call last):
File "mechanize\_urllib2_fork.py", line 1236, in do_open
File "http\client.py", line 1286, in request
File "http\client.py", line 1332, in _send_request
File "http\client.py", line 1281, in endheaders
File "http\client.py", line 1041, in _send_output
File "http\client.py", line 979, in send
File "http\client.py", line 1458, in connect
File "ssl.py", line 517, in wrap_socket
File "ssl.py", line 1108, in _create
File "ssl.py", line 1379, in do_handshake
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "runpy.py", line 198, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 215, in main
File "calibre\gui2\convert\gui_conversion.py", line 31, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1122, in run
File "calibre\customize\conversion.py", line 245, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 138, in convert
File "calibre\web\feeds\news.py", line 1069, in download
File "calibre\web\feeds\news.py", line 1248, in build_index
File "<string>", line 114, in parse_index
File "<string>", line 95, in parse_section
File "calibre\web\feeds\news.py", line 714, in index_to_soup
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 287, in _mech_open
File "mechanize\_opener.py", line 193, in open
File "mechanize\_urllib2_fork.py", line 425, in _open
File "mechanize\_urllib2_fork.py", line 414, in _call_chain
File "calibre\utils\browser.py", line 29, in https_open
File "mechanize\_urllib2_fork.py", line 1240, in do_open
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)>

[kovidgoyal]

You need to visit the website using Edge that will install the needed SSL certificate into Windows.

[Xer00]

Originally Posted by kovidgoyal

You need to visit the website using Edge that will install the needed SSL certificate into Windows.

Sorry, which website? The Guardian one in the code?

[Xer00]

Originally Posted by Xer00

Sorry, which website? The Guardian one in the code?

If so, i have just done that and unfortunately the same error is happening.

[unkn0wn]

def get_browser(self):
        return BasicNewsRecipe.get_browser(self, verify_ssl_certificates=False)

add this to the recipe