lemburg mbuegenixcom 20080422 1833 bill janssen wrote 2002 paper language character set determination method based ngram statistics izumi suzuki yoshiki mikami ario ohsato yoshihide chubachi seems pretty good way go thanks reference looks like existing research hasnt made mainstream yet heres current project httpwwwlanguageobservatoryorg looks like focusing language detection another interesting paper using ngrams language identification web pages bruno martins mário j silva httpxldbfculptdatapublications_attachngramarticlepdf one using compression text categorization using compression models eibe frank chang chui ian h witten httpportalacmorgcitationcfmid789742 theyre looking lses languagescriptencoding triples script way using particular character set write particular language system requirements r1 response must either correct answer unable detect unable detect includes registered registered set lses r2 applicable multilse texts r3 never accept wrong answer even program enough data lse r4 applicable lse text wrong answers biggest disadvantage would seem registration data particular lse kind bulky order 10000 shiftcodons three bytes 30k uncompressed httpportalacmorgft_gatewaycfmid772759typepdf server based application doesnt sound large unless youre using broad scope dont think youd need hundred lses typical application nothing youd want put python stdlib though bill imho research done area standard module added pythons stdlib knows perhaps lucky time everyone using utf8 anyway walked computational linguistics group asked often combined language guessing uses similar approach using characters instead bytes apparently usually done high confidence course theyre usually looking clean texts random stuff ill see get references report back research done 90s bill marcandre lemburg egenixcom professional python services directly source 1 apr 22 2008 pythonzope consulting support httpwwwegenixcom mxodbczopedatabaseadapter httpzopeegenixcom mxodbc mxdatetime mxtexttools httppythonegenixcom ________________________________________________________________________ try mxodbczopeda windowslinuxsolarismacosx free egenixcom software skills services gmbh pastorloehstr48 d40764 langenfeld germany ceo diplmath marcandre lemburg registered amtsgericht duesseldorf hrb 46611 _______________________________________________ pythondev mailing list zvlllneumpythonorg httpmailpythonorgmailmanlistinfopythondev unsubscribe httpmailpythonorgmailmanoptionspythondevpythondev40tangomucom fri 08 aug 2008 003821 0200 pythondev encoding detection standard library