Unicode Normalization Forms: When ö ≠ ö
A major problem with Unicode is that it gives you strange ideas about text: that you can somehow take human text encoded in Unicode and answer questions like &quot;how many letters does this have&quot; or &quot;are these two pieces of text different&quot; or &quot;split this text into words&quot; in a way that works generically for any langauge or context.<p>These are all myths, and APIs for such things are bugs. The only thing you can meaningfully do with two pieces of arbitrary Unicode text is to say if they are byte-by-byte equal. For any other operation, you need to have specific business logic.<p>For example, are &quot;Ionuț&quot; and &quot;Ionut&quot; and &quot;Ionutz&quot; the same string or different strings? There is no generic answer: depending on the intended business logic, they may be identical or not (e.g. if we consider these to be Romanian names, they should be considered identical for search purposes, but probably not identical for storage purposes, where you want to remember exactly how the person spelled their name).<p>A related problem is that most langauges have no separate types for Text&#x2F;String on one hand, and Symbol on the other. Text or other Strings should be opaque human text, that can only be interpreted by specific code, offering almost no API (only code point iteration). Symbols should be a restricted subset of Unicode that can offer fuller features, such as lengths, equality, separation into words etc. This would be the type of JSON key names used in serialization and deserialization, for example.