Making "invisible"characters visible?

[Phssthpok]

If I use characters such as ZWNJ or soft hyphens, I insert them as numeric entities &#xxx; in the text. However, the "smarten punctuation" tools replaces them by the actual characters, which are invisible. It is then impossible to find them and change them. My solution so far has been to leave out the hash marks (so "&xxx;" instead of "&#xxx;"), smarten the punctuation, and then re-insert the hash marks.

I have also come across similar problems in books which already contain invisible characters which mess up my formatting, and the only way I've found to deal with them is to paste the offending text into an old 8-bit-character editor (which displays any 16-bit characters as a question mark) so I can locate them, then go back to the Calibre editor and retype the text around the problem characters to delete them.

Would it be possible to change things somehow so that these "invisible" characters (which include others apart from zwnj and soft-hyphen, but I don't have a list) are always displayed as numeric entities in the editor to make them visible?

[kovidgoyal]

This is not something I am willing to do. Its adds a lot of overhead (entites are replaced by characters at parsing stage in calibre so pretty much every automated action/tool will have this effect) for everyone for a relatively uncommon use case. I suggest you just create a couple of saved search and replace expressions, that substitute the characters with the entities. Then you can go from character->entity at the click of a button or two.

[Phssthpok]

Originally Posted by kovidgoyal

This is not something I am willing to do. Its adds a lot of overhead (entites are replaced by characters at parsing stage in calibre so pretty much every automated action/tool will have this effect) for everyone for a relatively uncommon use case. I suggest you just create a couple of saved search and replace expressions, that substitute the characters with the entities. Then you can go from character->entity at the click of a button or two.

OK. This is easy enough for soft hyphens, but seems to be impossible for ZWNJ. I can enter both ZWNJ and soft hyphens into the search dialog from the "Insert character"dialog, but unlike soft hyphens, a search for a ZWNJ always fails. (It also leaves me with the mystery of what the other invisible characters are that mess up my formatting, but that's a separate problem.)

[kovidgoyal]

Using the Find expression

\u200c (regex mode)

works for me.

[Phssthpok]

Originally Posted by kovidgoyal

Using the Find expression

\u200c (regex mode)

works for me.

Ah. Of course. Thank you.

For anyone who's interested, the following regex-function replaces all matched characters with numeric entities:

def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    result = ''
    for c in match.group():
        result += '&#%d;' % ord(c)
    return result

Search for e.g. (\u200c|\u00ad) and "replace all" using this function.