02-17-2015, 03:33 AM | #1 |
Member
Posts: 11
Karma: 10
Join Date: Apr 2010
Device: Kindle
|
New York Times Recipe failing to verify SSL Cert
Using the latest calibre (2.19.0) on Fedora 21 (official release, not distro package), the NYTimes recipe is failing to fetch with an SSL error. The debug log is below.
Recent SSL + python related bugs in other apps were fixed by installed the python-service-identity package (or service_identity on pypi), but this doesn't seem to be related. Verifying the domain manually works with openssl works fine: Code:
$ openssl s_client -connect myaccount.nytimes.com:443 <snip> Verify return code: 0 (ok) Code:
calibre, version 2.19.0 (linux2, isfrozen: True) Conversion Error: Failed: Fetch news from New York Times Fetch news from New York Times Resolved conversion options calibre version: 2.19.0 {'asciiize': False, 'author_sort': None, 'authors': None, 'base_font_size': 0, 'book_producer': None, 'change_justification': 'original', 'chapter': None, 'chapter_mark': 'pagebreak', 'comments': None, 'cover': None, 'debug_pipeline': None, 'dehyphenate': True, 'delete_blank_paragraphs': True, 'disable_font_rescaling': False, 'dont_compress': False, 'dont_download_recipe': False, 'duplicate_links_in_toc': False, 'embed_all_fonts': False, 'embed_font_family': None, 'enable_heuristics': False, 'expand_css': False, 'extra_css': None, 'extract_to': None, 'filter_css': None, 'fix_indents': True, 'font_size_mapping': None, 'format_scene_breaks': True, 'html_unwrap_factor': 0.4, 'input_encoding': None, 'input_profile': <calibre.customize.profiles.InputProfile object at 0x7f3fdbec1c50>, 'insert_blank_line': False, 'insert_blank_line_size': 0.5, 'insert_metadata': False, 'isbn': None, 'italicize_common_cases': True, 'keep_ligatures': False, 'language': None, 'level1_toc': None, 'level2_toc': None, 'level3_toc': None, 'line_height': 0, 'linearize_tables': False, 'lrf': False, 'margin_bottom': 5.0, 'margin_left': 5.0, 'margin_right': 5.0, 'margin_top': 5.0, 'markup_chapter_headings': True, 'max_toc_links': 50, 'minimum_line_height': 120.0, 'mobi_file_type': 'old', 'mobi_ignore_margins': False, 'mobi_keep_original_images': False, 'mobi_toc_at_start': False, 'no_chapters_in_toc': False, 'no_inline_navbars': True, 'no_inline_toc': False, 'output_profile': <calibre.customize.profiles.KindlePaperWhiteOutput object at 0x7f3fdbec63d0>, 'page_breaks_before': None, 'personal_doc': '[PDOC]', 'prefer_author_sort': False, 'prefer_metadata_cover': False, 'pretty_print': False, 'pubdate': None, 'publisher': None, 'rating': None, 'read_metadata_from_opf': None, 'remove_fake_margins': True, 'remove_first_image': False, 'remove_paragraph_spacing': False, 'remove_paragraph_spacing_indent_size': 1.5, 'renumber_headings': True, 'replace_scene_breaks': '', 'search_replace': None, 'series': None, 'series_index': None, 'share_not_sync': False, 'smarten_punctuation': False, 'sr1_replace': '', 'sr1_search': '', 'sr2_replace': '', 'sr2_search': '', 'sr3_replace': '', 'sr3_search': '', 'start_reading_at': None, 'subset_embedded_fonts': False, 'tags': None, 'test': False, 'timestamp': None, 'title': None, 'title_sort': None, 'toc_filter': None, 'toc_threshold': 6, 'toc_title': None, 'unsmarten_punctuation': False, 'unwrap_lines': True, 'use_auto_toc': False, 'verbose': 2} InputFormatPlugin: Recipe Input running Using custom recipe Traceback (most recent call last): File "site.py", line 51, in main File "site-packages/calibre/utils/ipc/worker.py", line 193, in main File "site-packages/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert File "site-packages/calibre/ebooks/conversion/plumber.py", line 1041, in run File "site-packages/calibre/customize/conversion.py", line 241, in __call__ File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 116, in convert File "site-packages/calibre/web/feeds/news.py", line 887, in __init__ File "<string>", line 391, in get_browser File "site-packages/mechanize/_mechanize.py", line 203, in open File "site-packages/mechanize/_mechanize.py", line 230, in _mech_open File "site-packages/mechanize/_opener.py", line 204, in open File "site-packages/mechanize/_urllib2_fork.py", line 457, in http_response File "site-packages/mechanize/_opener.py", line 221, in error File "site-packages/mechanize/_urllib2_fork.py", line 332, in _call_chain File "site-packages/mechanize/_urllib2_fork.py", line 571, in http_error_302 File "site-packages/mechanize/_mechanize.py", line 203, in open File "site-packages/mechanize/_mechanize.py", line 230, in _mech_open File "site-packages/mechanize/_opener.py", line 193, in open File "site-packages/mechanize/_urllib2_fork.py", line 344, in _open File "site-packages/mechanize/_urllib2_fork.py", line 332, in _call_chain File "site-packages/mechanize/_urllib2_fork.py", line 1170, in https_open File "site-packages/mechanize/_urllib2_fork.py", line 1118, in do_open URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)> |
02-17-2015, 08:31 AM | #2 |
creator of calibre
Posts: 44,145
Karma: 22670164
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
You are missing the root certificate needed to verify that site. Install/update your distros ca-certificates package.
|
Advert | |
|
02-19-2015, 05:55 AM | #3 |
Member
Posts: 11
Karma: 10
Join Date: Apr 2010
Device: Kindle
|
Thanks for responding Kovid.
I reinstalled my distros ca-certificates (which uses mozilla's bundle btw). I definitely have the root certificate. The issue is in fact that calibre, or a library, is looking in the wrong place for the cert bundle. But first let me demonstrate I do in fact have the root CA's cert. Using Chrome, which bundles its own certs, I see the chain as: Code:
1. GeoTrust Global CA DE 28 F4 A4 FF E5 B9 2F A3 C5 03 D1 A3 49 A7 F9 96 2A 82 12 2. RapidSSL CA C0 39 A3 26 9E E4 B8 E8 2D 00 C5 3F A7 97 B5 A1 9E 83 6F 47 3. *.nytimes.com DB 76 F2 CF 5F A4 05 5E D2 95 63 6E 6A 8D 5F 6A 66 D9 54 56 all fingerprints SHA1 Code:
$ awk -v cmd='openssl x509 -noout -fingerprint' ' /BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-bundle.trust.crt | grep DE:28:F4:A4:FF:E5:B9:2F:A3:C5:03:D1:A3:49:A7:F9:96:2A:82:12 -B1 SHA1 Fingerprint=DE:28:F4:A4:FF:E5:B9:2F:A3:C5:03:D1:A3:49:A7:F9:96:2A:82:12 Looking in the wrong place I used strace to that the CA cert bundle isn't being accessed. using Code:
$ strace -e open,access ebook-convert nytimes.recipe foo.mobi --test --username XXX --password ZZZ &> strace.log Code:
open("/opt/calibre/lib/python2.7/site-packages/calibre/urllibmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory) open("/opt/calibre/lib/python2.7/site-packages/calibre/urllib.py", O_RDONLY) = -1 ENOENT (No such file or directory) open("/opt/calibre/lib/python2.7/site-packages/calibre/urllib.pyo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 3 open("/etc/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3 open("/etc/host.conf", O_RDONLY|O_CLOEXEC) = 3 open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3 open("/opt/calibre/lib/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3 open("/opt/calibre/lib/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = 3 open("/opt/calibre/lib/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 3 Traceback (most recent call last): File "site.py", line 51, in main File "site-packages/calibre/ebooks/conversion/cli.py", line 360, in main File "site-packages/calibre/ebooks/conversion/plumber.py", line 1041, in run File "site-packages/calibre/customize/conversion.py", line 241, in __call__ File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 116, in convert File "site-packages/calibre/web/feeds/news.py", line 887, in __init__ File "<string>", line 391, in get_browser File "site-packages/mechanize/_mechanize.py", line 203, in open File "site-packages/mechanize/_mechanize.py", line 230, in _mech_open File "site-packages/mechanize/_opener.py", line 193, in open File "site-packages/mechanize/_urllib2_fork.py", line 344, in _open File "site-packages/mechanize/_urllib2_fork.py", line 332, in _call_chain File "site-packages/mechanize/_urllib2_fork.py", line 1170, in https_open File "site-packages/mechanize/_urllib2_fork.py", line 1118, in do_open URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)> open("/opt/calibre/lib/python2.7/site-packages/calibre/shutil.so", O_RDONLY) = -1 ENOENT (No such file or directory) open("/opt/calibre/lib/python2.7/site-packages/calibre/shutilmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory) open("/opt/calibre/lib/python2.7/site-packages/calibre/shutil.py", O_RDONLY) = -1 ENOENT (No such file or directory) open("/opt/calibre/lib/python2.7/site-packages/calibre/shutil.pyo", O_RDONLY) = -1 ENOENT (No such file or directory) +++ exited with 1 +++ Symlinking the bundle to /etc/ssl/cert.pem, and running calibre fixes the validation error. However this isn't a good solution nor workaround. I assume the bug is in python, not in calibre proper. But could you add additional cert path search locations? Here is a simple proof of concept: https://gist.github.com/Ramblurr/bf48299caaadeb17d392 Last edited by Ramblurr; 02-19-2015 at 06:01 AM. Reason: fixed poc url |
02-19-2015, 07:05 AM | #4 |
creator of calibre
Posts: 44,145
Karma: 22670164
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Sigh, I love linux distros. Neither calibre nor python hardcode any paths to ssl certs. Those come from the openssl library. See the functions X509_get_default_cert_file_env and X509_get_default_cert_file
IIRC those in turn can be controlled via environment variables SSL_CERT_FILE and SSL_CERT_DIR Presumably, your distro patches its build of openssl to not use the default path. However, calibre comes with its own bundled dependencies, including the openssl libs, which will therefore not have your distro specific patch. So they will not be able to find the files. Whatever distro you are using should either set the environment variables, or the symlinks, or better yet stop patching upstream packages willy nilly. Since I doubt your distro is likely to see reason, your remaining workarounds are to either set those env vars yourself, use symlinks, or delete the bundled openssl libs in the calibre package, which will then cause it to use the distro versions (assuming the C runtimes are ABI compatible). Last edited by kovidgoyal; 02-19-2015 at 07:20 AM. |
02-20-2015, 04:32 AM | #5 |
Member
Posts: 11
Karma: 10
Join Date: Apr 2010
Device: Kindle
|
Yea, distro standards are generally a mess. Each distro has "their way" and if you deviate you're in for a world of hurt.
Fedora's "way" in this case is simply they expect you to rely on their system packages, bundling everything separate is anathame (in their eyes). Of course they are so slow at pushing updates to calibre that we want a separately bundled app anyways. I might start building calibre from the src rpm and just updating it myself. Anyways if you're interested in a workaround: 1. Replacing the bundled libs with symlinks On Fedora 21 the ABIs are compatible, for now. In /opt/calibre/lib symlinked as follows. Code:
libcrypto.so.1.0.0 -> /usr/lib64/libcrypto.so libssl.so.1.0.0 -> /usr/lib64/libssl.so.10 Launching calibre manually with "env SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt calibre" works, or you can edit the .desktop files (/usr/local/share/applications/calibre-*desktop) to set the variable yourself. Option one is easiest for me, and I'll be doing that until I take the time to roll my own RPM of calibre. Kovid, any chance calibre could detect the location? According to the article below, the best solution is Code:
1. Check if /etc/pki/tls/certs/ca-bundle.crt exists, if so use with SSL_CERT_FILE [Fedora, Redhat, Arch etc] 2. Check if /etc/ssl/certs exists, if so use with SSL_CERT_DIR [Debian, Ubuntu, etc] 3. Fall back on default settings A note about SSL/TLS trusted certificate stores, and platforms (OpenSSL and GnuTLS) - https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/ |
Advert | |
|
02-20-2015, 05:33 AM | #6 |
creator of calibre
Posts: 44,145
Karma: 22670164
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
That should do the trick https://github.com/kovidgoyal/calibr...e7e57250bce8a5
I haven't tested it on a Fedora machine since I dont want to build a VM just for this, let me know if it doesn't work in the next release. |
02-27-2015, 04:31 AM | #7 |
Member
Posts: 11
Karma: 10
Join Date: Apr 2010
Device: Kindle
|
This is fixed for me in v.2.20. Thanks Kovid.
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
New York Times Recipe | dieterpops | Recipes | 1 | 01-20-2013 12:26 PM |
Which New York Times recipe? | jdomingos76 | Recipes | 1 | 03-25-2011 08:40 PM |
Help - New York Times Recipe | brutalist | Recipes | 6 | 03-20-2011 10:17 PM |
New York Times recipe broken? | gianfri | Calibre | 1 | 03-20-2010 09:52 AM |
New York Times recipe | madrone26 | Calibre | 4 | 04-02-2009 01:13 PM |