UTF-8 String Indexing Strategies
Is there any real case where code point indexing is useful? It seems like all these attempts to restrict strings in such a way to accommodate code points is just introducing complexity with no gain.<p>UTF-8 was designed to be an encoding (of code points) on top of the &quot;bytes&quot; abstraction just as Unicode is designed to be an encoding (of human text) on top of the &quot;code points&quot; abstraction. I think it should be uncontroversial that there are very good reasons to at least handle arbitrary sequences of code points (eg, you want to be able to handle input from future versions of Unicode, and you don&#x27;t know about the grapheme clustering of those code points), but I don&#x27;t see a good reason <i>not</i> to handle arbitrary sequences of bytes.<p>The only reason I can see is in ensuring that text is losslessly convertible to other UTFs, particularly UTF-16 (which exists for historical reasons), but this just seems like a matter of <i>when</i> the information is lost (is it during conversion from string to UTF-16, or from bytes to string), not <i>if</i> it is lost.<p>As far as I can tell with the Python story, for example, people decided to add special &quot;Unicode&quot; strings into Python 2, then presumably some code used the &quot;Unicode&quot; strings and some code used the &quot;byte&quot; strings, so this situation is obviously underisable .. then in Python 3 they tried fixing it by replacing which sort of string was the default. Why would it not have been better to just improve Unicode support for the existing strings instead of splitting the type into two and forcing everyone to decide whether their strings are for &quot;bytes&quot; or for &quot;Unicode&quot;?