How Python does Unicode
UCS-4 is essentially never the right choice.  It wastes space and thus messes up your cache.  UCS-2 can be the right choice if the language you're encoding uses a lot of non-Latin glyphs (i.e. East Asian languages) but suffers from the same problem as UCS-2.  UTF-8 is a good default: for most strings it's very compact, and for strings with a lot of multibyte codepoints it doesn't compare too unfavorably with UTF-16.Python 3 tried to have its cake and eat it too by choosing the most compact encoding depending on the string, but in practice this wastes a lot of space.  You'll double (or heaven forbid quadruple) your string size because of a single codepoint, and these codepoints are almost always a small percentage of the string.  That's actually why UTF-16 and UTF-8 exist.It would have been better for strings to default to UTF-8, and to add an optional encoding so the programmer can specify what kind of encoding to use.  As it is now, in order to use (for example) UTF-16 strings in Python you have to keep them around as bytes, decode them to a string, perform string operations, and reencode them to bytes again.  Any benefit you get from using UTF-16 vanishes the moment you need to operate on it like a string, in other words.I get that the idea was to maintain indexing via codepoint, but (again) in practice that's not great: usually you want to index via grapheme -- if you want to index at all.A better solution is to allow programmers to specify string encoding and default it to UTF-8.  From that, there's a clear path to everything you'd want to do.