<i>What if I so happen to be working with Chinese Unicode files without BOM that are also valid ASCII files?</i><p>What&#x27;s the longest natural Chinese substring you can find for which this is the case?  When we ran our encoding-and-language-detection heuristic developed for our research work I think, and this is 10+ years ago, it was 5 characters long -- there was no substring 6+ characters (12 bytes of UTF-16) which could be reasonably written in ASCII.<p>This is because much of ASCII is unprintable, more-than-doubly so if you&#x27;re concerned strictly with 7-bit ASCII as opposed to e.g. Latin-1.  When&#x27;s the last time you saw a string contain 0x18 (&quot;cancel&quot;), 0x07 (&quot;bell&quot;), etc?<p>An assortment of 0x..18 characters for your amusement: 予,先,券,単,又.  I picked ones which are included in commonly used words in Japanese.  Seeing any one of these once in UTF-16 is dispositive that the bytestream is not ASCII.<p>This problem is worth thinking about (which is why a customer asked a team of CS researchers and linguists to think about it, including one barely competent programmer who nonetheless had reasonably good intuitions for what character distributions looked like), but it turns out to be much, much less hard than many people originally expected.<p>Anyhow, long story short: make a histogram of the bytes, dot product with the histogram signature you have of a few large corpora in decent-guess-based-on-apriori-knowledge languages&#x2F;encoding, normalize.  The screamingly obvious candidate is almost always the winner.<p>If you want to do it really, really quickly you can even evaluate this heuristic with a Bloom filter in an FPGA.