What&#x27;s hard, however, is telling multi-byte CJK encodings apart, and telling them apart from UTF-8.<p>I&#x27;ve tried it, as I was working on extending my mojibake-fixing library [1] to the language that gave us the word &quot;mojibake&quot;. The ambiguous cases <i>actually happen</i>.<p>[1] <a href="http:&#x2F;&#x2F;github.com&#x2F;LuminosoInsight&#x2F;python-ftfy" rel="nofollow">http:&#x2F;&#x2F;github.com&#x2F;LuminosoInsight&#x2F;python-ftfy</a><p>Suppose you found this string (which someone actually tweeted) encoded in Shift-JIS:<p><pre><code>    (|| * m *)ｳ､ｳｯﾌﾟ･･
</code></pre>
Those half-width katakana with a comma in the middle look a lot like typical Japanese mojibake, so the classifier would be entirely justified in deciding that Shift-JIS is <i>wrong</i>. Next let&#x27;s try decoding those bytes as EUC-JP instead:<p><pre><code>    (|| * m *)海劾餅ゥ
</code></pre>
Hey, maybe that&#x27;s a name or something! No other encoding works, so the classifier happily decides it&#x27;s EUC-JP. Except this string is complete nonsense and the person actually meant the first string.<p>I do agree with you that it&#x27;s a problem worth trying to solve (clearly). We can&#x27;t just throw up our hands and say &quot;this is a problem that shouldn&#x27;t happen, so let&#x27;s not solve it&quot;. Microsoft Office exists, and its mojibake-causing &quot;features&quot; will never be removed due to backward compatibility, so this happens and will continue to happen.<p>But, any ideas where the corpus for telling apart encodings should actually come from? I&#x27;ve been using Twitter, but this limits the domain. Only thoroughly defective Twitter clients send anything but UTF-8 to Twitter. There are lots of defective Twitter clients, it turns out, but the creative mistakes that Excel users make every day are one in a billion on Twitter. I can apply artificial mojibake, but then I&#x27;m not correlating the text with the encoding it&#x27;s likely to be in.