bill janssen olchxynparccom 2002 paper language character set determination method based ngram statistics izumi suzuki yoshiki mikami ario ohsato yoshihide chubachi seems pretty good way go theyre looking lses languagescriptencoding triples script way using particular character set write particular language system requirements r1 response must either correct answer unable detect unable detect includes registered registered set lses r2 applicable multilse texts r3 never accept wrong answer even program enough data lse r4 applicable lse text wrong answers biggest disadvantage would seem registration data particular lse kind bulky order 10000 shiftcodons three bytes 30k uncompressed httpportalacmorgft_gatewaycfmid772759typepdf bill imho research done area standard module added pythons stdlib knows perhaps lucky time everyone using utf8 anyway walked computational linguistics group asked often combined language guessing uses similar approach using characters instead bytes apparently usually done high confidence course theyre usually looking clean texts random stuff ill see get references report back research done 90s bill _______________________________________________ pythondev mailing list zvlllneumpythonorg httpmailpythonorgmailmanlistinfopythondev unsubscribe httpmailpythonorgmailmanoptionspythondevpythondev40tangomucom thu 07 aug 2008 151249 0700 pythondev encoding detection standard library