function mode: surprising count

[roger64]

Hi

I tried some functions within an EPUB. One which is included in the help file (hyphen) and another which was custom-built for me by a MR Spanish friend. The two are working very fine and are quite helpful.

However, they seem to suffer from the same small defect: when the function has been done, the Calibre editor displays a small window announcing the job done, and offers to show me where the replacements did take place. It also announces the number of occurrences replaced. This is here where the "error" appears.

I found each time for each function that the reported number of occurrences was wrong. In both cases, it looks like the counting is out of touch with reality.
- For the hyphens, it reported over 2000 occurrences replaced when it really replaced about 30 (the text had been proof-read and I did introduce voluntary mistakes).
- For the other function, it consistently reported 1386 occurrences (even if the search was expanded), but when I checked later with a plain regex it did in fact replace 1802 occurrences.

[kovidgoyal]

I need an example file and find expression and function with which to reproduce the issue.

[roger64]

OK I'll do that but maybe tomorrow. Sorry

[roger64]

Hi

I had to prepare it for another book. I join a reduced version of a French book. As you can check first, it contains now:
- 0 nnbsp (\u202F)
- 371 hyphen-minus (-) + 10 wrong ones set in chapter2: total: 381

1. - The first function is about nnbsp
search:

>[^\n<]*?<

function text:

def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    return match.group().replace("'","’").replace(">— ",">—@").replace(">—",">—@").replace(" !","@!").replace("!","@!").replace(" ?","@?").replace("?","@?").replace(" ;","@;").replace(";","@;").replace(" :","@:").replace(":","@:").replace("« ","«@").replace("«","«@").replace(" »","@»").replace("»","@»").replace("@@","@").replace("@","\u202f")

It announces for me: 536 replacements done
But if you check with a regex there are now 650 nnbsp.

2. - The second function is hyphen (coupled with French dictionary)

search:

>[^<>]+<

function text

import regex
from calibre import replace_entities
from calibre import prepare_string_for_xml

def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):

    def replace_word(wmatch):
        # Try to remove the hyphen and replace the words if the resulting
        # hyphen free word is recognized by the dictionary
        without_hyphen = wmatch.group(1) + wmatch.group(2)
        if dictionaries.recognized(without_hyphen):
            return without_hyphen
        return wmatch.group()

    # Search for words split by a hyphen
    text = replace_entities(match.group()[1:-1])  # Handle HTML entities like &
    corrected = regex.sub(r'(\w+)\s*-\s*(\w+)', replace_word, text, flags=regex.VERSION1 | regex.UNICODE)
    return '>%s<' % prepare_string_for_xml(corrected)  # Put back required entities

After it ran, it reports 996 replacements.
But only 15 were done: the 10 wrong in chapter 2 have been corrected and the following 5: frou-frou(2), en-tête, par-dessus, porte-manteaux. These last five should not have been corrected but this is another story.

[kovidgoyal]

Ah I see your source of confusion. The number reported is the number of locations in the text that matched the search expression, not the number of replacements. There is no way to know what replacement means in function mode.

So if your search expression matches 10 places int he text and in each place your function does 2 replacements inside the matched text, the number reported will be 10, not 20.

Even in a normal search and replace, for example if you search for

a

and replace with

a

The number reported will be some thousands, but the number of actual replacements will be zero, because replacing a by a does not actually change anything.

[roger64]

Thanks for your explanation. Let's say that I have been confused by a confusing report

What's the purpose of reporting such a - huge - number if it's useless?

I understand that if I replace a with a, I will get an absurd result to an absurd question. But this is not the same as here. Furthermore "remplacé 1 occurrence" can just be understood that one occurrence has been replaced. And, if you deal with large numbers, there is no way to check easily that the reporting figure does not relate at all to the effective amount of replacements.

[kovidgoyal]

Answer me this:

If you replace abc with xby have you made one replacement or two? i.e. have you replaced a with x and c with y or have you replaced abc with xby?

Now look back at what I said, it is not possible to know how many replacements are performed in function mode and the number is the number of matches of the search expression. You should be able to connect the rest of the dots yourself.

[roger64]

I hope I get it right this time: as far as functions are concerned, displayed search matches figures like the one in the screenshot above bear no relation to the number of replacements that the user is looking for using this function.

Thank you for providing such a powerful tool.