Replacement of Replacement Character

[Leonatus]

Once that I'm about to adjust my news download, I've still got a tiny little question: My news have in the online original quotation marks of this sort:

„...“

.
In the downloaded news they are replaced by the replacement character:

�...�

.
No big problem, but ... ugly.
Is it possible to edit the recipe in a way that replaces the replacement characters by quotation marks (of any kind)?
The original site is encoded in ISO-8859-1, and so is the encoding of the recipe. I replaced it by utf-8, but this didn't help.

[kovidgoyal]

Make sure the encoding field in the recipe matches the encoding of the website and you will be fine. if you want to do search and replace in the recipe you can use preprocess_regexps

[Leonatus]

This has been the first thing I've been trying in spite of my technical ignorance: to check if the encoding of the original website where the news is from corresponded to the encoding of the recipe - and to my astonishment it did. So this is not the culprit, as it seems.

How do I use preprocess_regexps "step by step", please (for I'm really technically ignorant, sorry)?

Edit: In the mean time I noticed that in single articles the quotation marks are displayed correctly, maintaining the same source code as the other articles. Hm .. the thing becomes interesting.

Edit': There is one difference, however: In the articles with replacement character, quotes are represented by „...“, whereas in the correctly dispayed articles they are "...".

[Leonatus]

I read in Calibre's documentation that the preprocess_regexps should look like that:

preprocess_regexps = [
   (re.compile(r'<!--Article ends here-->.*</body>', re.DOTALL|re.IGNORECASE),
    lambda match: '</body>'),
]

Unfortunately, I have no idea how to progreed in order replace all „ and “ by ". Could one of the pros here give me, please, a hint how to do this?

[siebert]

Untested:

preprocess_regexps = [
   (re.compile(r'[„“]'),
    lambda match: '"'),
]

[Leonatus]

Thank you, but doesn't work. The replacement characters still appear.

[siebert]

I don't think I ever used unicode in regular expressions. Did you just copy my code or did you try to replace the „“ chars in it with the ones copied from the source webpage?

Otherwise this variant might work better:

preprocess_regexps = [
   (re.compile(r'„|“'),
    lambda match: '"'),
]

Or you could post the whole recipe here, so I can test it.

[Leonatus]

The variant didn't work either. I had simply copy/pasted the code fromyour post, the characters reproduced in #1 beeing originally copied from the website resp. the ebook-viewer of Calibre (the display is the same as on my reader).
The recipe is originally this:

from calibre.web.feeds.news import BasicNewsRecipe


class AdvancedUserRecipe1295262156(BasicNewsRecipe):
    title = u'kath.net'
    __author__ = 'Bobus'
    description = u'Katholische Nachrichten'
    oldest_article = 7
    language = 'de'
    max_articles_per_feed = 100
    no_stylesheets = True
    encoding = 'iso-8859-1'

    feeds = [(u'kath.net', u'https://www.kath.net/2005/xml/index.xml')]

    def print_version(self, url):
        return url + "/print/yes"

    def get_browser(self, *a, **kwargs):
        kwargs['verify_ssl_certificates'] = False
        return BasicNewsRecipe.get_browser(self, *a, **kwargs)

    extra_css = 'td.textb {font-size: medium;}'

thank you for testing!

[siebert]

Sorry, all the things I googled and tried didn't work. I'm running out of ideas.

[kovidgoyal]

you need to replace the replacement character, not the quote, since the quote will already have been repaced by the replacement character at the time preprocess_regexp runs

[Leonatus]

Originally Posted by kovidgoyal

you need to replace the replacement character, not the quote, since the quote will already have been repaced by the replacement character at the time preprocess_regexp runs

Hm, that has been my consideration, too, but it didn't work either at least following Siebert's suggestion. Anyway, thanks for the help!

[Leonatus]

Should I perhaps escape the replacement character, and how do I do this?

[theducks]

Originally Posted by Leonatus

Should I perhaps escape the replacement character, and how do I do this?

the backslash is the 'escape'. \\ allows the \ to be the target.
in theory you could escape any character \e\s\c\a\p\e
(if in doubt, I escape symbols I search for. Not all, really need to be escaped)

[Leonatus]

Originally Posted by theducks

the backslash is the 'escape'. \\ allows the \ to be the target.
in theory you could escape any character \e\s\c\a\p\e
(if in doubt, I escape symbols I search for. Not all, really need to be escaped)

I did this, but at no avail. My thought now is that perhaps the ISO 8859-1 code for the replacement character should be searched for, but this is very much beyond my capacities.
Edit: In Wikipedia Specials (Unicode block) I found this: "... It has become increasingly common for software to interpret invalid UTF-8 by guessing the bytes are in another byte-based encoding such as ISO-8859-1."

[lui1]

According to wikipedia (see ISO-8859-1 and Windows-1252) webpages and emails are commonly mislabeled with the encoding ISO-8859-1 when it should be Windows-1252. Most web browsers and email clients will treat this encoding as Windows-1252. This practice is so prevalent that it became part of the HTML5 specification. So any webpage which claims to be encoded with ISO-8859-1 should be treated as being encoded with Windows-1252.

encoding = 'windows-1252'