The UTF-8-Everywhere Manifesto
tl:dr; Use UTF-8 when you need to use unicode with legacy APIs, never anywhere else.<p>UNIX isn't UTF-8 because UTF-8 is better, UNIX is UTF-8 because you can pass UTF-8 strings to functions that expect ASCII and it kinda works. This is really the only thing you need to know about UTF-8 and why it's better.<p>There are few pieces of software that don't have to talk to legacy APIs that store strings natively in UTF-8.<p>C# and Java are probably the best examples of software that was engineered from the ground up and thus uses UTF-16 internally because it's much less likely to run into issues like String.length returning 32 yet only containing 31 characters. If you use UTF-8 expect this result anytime a string contains a real genuine apostrophe.<p>"UTF-8 and UTF-32 result the same order when sorted lexicographically. UTF-16 does not."<p>This is complete and utter bullshit, to sort a string lexicographically you need to decode it, if you've decoded the string into UNICODE then they sort the exact same way.<p>There are lots of gotchas for sorting UNICODE strings including normalization because you can write the semantically equivalent strings in unicode multiple ways. eg. ligatures.<p>If you're sorting bit strings that happen to contain UTF-8/32 then you're not sorting lexicographically and your results will be screwed up anyway.