lemburg mbuegenixcom 20080421 2331 martin v löwis wrote useful get hunk data _should_ sort intelligible text big scary internet say posted web form email message want something useful say search content dont think part standard library people mistake tells certain 1 also think better educate people add correct encoding information text data rather give guess mechanism httpchardetfeedparserorgdocsfaqhtmlfaqyippie chardet based mozilla algorithm least experience algorithm doesnt work well mozilla algorithm may work asian encodings due fact encodings usually also bound specific language use character word frequency analysis encodings encode far single language eg utf8 latin1 correct detection rate rather low problem becomes completely even difficult leaving normal text domain mixing languages text eg trying detect source code comments using nonascii encoding trick pass text codec see whether roundtrips also doesnt necessarily help latin1 example always roundtrip since latin1 subset unicode imho research done area standard module added pythons stdlib knows perhaps lucky time everyone using utf8 anyway marcandre lemburg egenixcom professional python services directly source 1 apr 22 2008 pythonzope consulting support httpwwwegenixcom mxodbczopedatabaseadapter httpzopeegenixcom mxodbc mxdatetime mxtexttools httppythonegenixcom ________________________________________________________________________ try mxodbczopeda windowslinuxsolarismacosx free egenixcom software skills services gmbh pastorloehstr48 d40764 langenfeld germany ceo diplmath marcandre lemburg registered amtsgericht duesseldorf hrb 46611 _______________________________________________ pythondev mailing list zvlllneumpythonorg httpmailpythonorgmailmanlistinfopythondev unsubscribe httpmailpythonorgmailmanoptionspythondevpythondev40tangomucom fri 08 aug 2008 000620 0200 pythondev encoding detection standard library