jeanpaul calderone mhmhlmqdivmodcom mon 21 apr 2008 175043 0100 michael foord wrote iybzpoboxcom wrote david sort text encoding detection module david standard library reason add david one theres suspect fact cant correctly determine encoding chunk text 100 time mitigates approach know heuristic based approach eg httpwwwvoidspaceorgukpythonarticlesguessing_encodingshtml borrowed docutils first place isnt approach although youre right general rely heuristics see charset detection features icu httpwwwicuprojectorguserguidecharsetdetectionhtml think osafs pyicu exposes apis httppyicuosafoundationorg jeanpaul _______________________________________________ pythondev mailing list zvlllneumpythonorg httpmailpythonorgmailmanlistinfopythondev unsubscribe httpmailpythonorgmailmanoptionspythondevpythondev40tangomucom thu 07 aug 2008 172616 0400 pythondev encoding detection standard library