ALERT: Potential Issues with Sigil 2.2.X and rtl languages and Normalization Forms - Page 3

KevinH · Yesterday, 05:07 PM

The unicode standard is not carved in stone and is in its 15th revision. Would you believe that new emojiis are one of its driving issues.

And AI and text analytics are some of the areas pushing most strongly for search ability and a using precomposed forms.

democrite · Yesterday, 05:14 PM

I can understand the usefulness for most users. It just seems there are enough potential issues with some languages, perhaps potentially more for non-Western, partially given the opinion of some it seems of Unicode in general, that I strongly feel it should be a setting and not something strictly enforced.

KevinH · Yesterday, 05:42 PM

Understood. As I said, given demand, I will create an environment variable to disable it once we get things working completely.

democrite · Yesterday, 05:49 PM

Great. I wasn't sure if the variable was either possibly planned or certain. Thank you very much.

democrite · Today, 02:38 PM

I’m not that familiar but I mention such since it seems to be relevant here and it seems that Sigil and some eReaders may not be supporting such.

Is canonical Unicode equivalence relevant here? Not necessarily storing the text in some normalized form, but it seems maybe the spec requires that apps normalize text just for the purpose of processing? So it seems maybe search should function the same way in apps as required by such?

KevinH · Today, 05:00 PM

Easy to do when processing an external file but any position information of a match location may then not match the original file because of combined characters shrinking length of its string representation. This makes replacement a big issue.

This is even more of a problem when done inside an editor that does not force normalize the strings in advance. Any match information (position and if match or not) will be incorrect unless underlying file is properly normalized the same way as the search string. This makes search and replacement impossible.

Yesterday, 05:07 PM	#31
KevinH Sigil Developer Posts: 7,803 Karma: 5446592 Join Date: Nov 2009 Device: many	The unicode standard is not carved in stone and is in its 15th revision. Would you believe that new emojiis are one of its driving issues. And AI and text analytics are some of the areas pushing most strongly for search ability and a using precomposed forms. Last edited by KevinH; Yesterday at 05:09 PM.

Yesterday, 05:49 PM	#34
democrite Evangelist Posts: 435 Karma: 77256 Join Date: Sep 2011 Device: none	ALERT: Potential Issues with Sigil 2.2.X and rtl languages and Normalization Forms Great. I wasn't sure if the variable was either possibly planned or certain. Thank you very much.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Potential Issues Upgrading? from 3.21 :O	rebeltaz	Calibre	22	03-20-2022 11:19 PM
New to Using Sigil and Having Issues	jester1972	Sigil	20	04-30-2017 09:24 AM
New Issues in Sigil 0.9.3	jafprrr	Sigil	11	03-10-2016 11:59 AM
issues with sigil 0.8.4	eregs	Sigil	2	02-27-2015 08:01 AM
Support for RTL Languages	Gonidae	Calibre	1	10-05-2012 05:13 AM

Yesterday, 05:14 PM	#32
democrite Evangelist Posts: 435 Karma: 77256 Join Date: Sep 2011 Device: none	I can understand the usefulness for most users. It just seems there are enough potential issues with some languages, perhaps potentially more for non-Western, partially given the opinion of some it seems of Unicode in general, that I strongly feel it should be a setting and not something strictly enforced.

Yesterday, 05:42 PM	#33
KevinH Sigil Developer Posts: 7,803 Karma: 5446592 Join Date: Nov 2009 Device: many	Understood. As I said, given demand, I will create an environment variable to disable it once we get things working completely.

Today, 02:38 PM	#35
democrite Evangelist Posts: 435 Karma: 77256 Join Date: Sep 2011 Device: none	I’m not that familiar but I mention such since it seems to be relevant here and it seems that Sigil and some eReaders may not be supporting such. Is canonical Unicode equivalence relevant here? Not necessarily storing the text in some normalized form, but it seems maybe the spec requires that apps normalize text just for the purpose of processing? So it seems maybe search should function the same way in apps as required by such?

Today, 05:00 PM	#36
KevinH Sigil Developer Posts: 7,803 Karma: 5446592 Join Date: Nov 2009 Device: many	Easy to do when processing an external file but any position information of a match location may then not match the original file because of combined characters shrinking length of its string representation. This makes replacement a big issue. This is even more of a problem when done inside an editor that does not force normalize the strings in advance. Any match information (position and if match or not) will be incorrect unless underlying file is properly normalized the same way as the search string. This makes search and replacement impossible.