Ftfy – fix Unicode that's broken in various ways
This could be a useful web service <i>for interactive use</i>, since all strings will be manually verified.<p>The underlying ftfy library was previously discussed three years ago (<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=8187418" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=8187418</a>) and my comments at the time are still relevant:<p>I can’t help but think that this [library] merely gives people the excuse they need for not understanding this “Things-that-are-not-ASCII” problem. Using this library is a desperate attempt to have a just-fix-it function, but it can never cover all cases, and will inevitably corrupt data. To use this library is to remain an ASCII neanderthal, ignorant of the modern world and the difference of text, bytes and encodings.<p>Let me explain in some detail why this library is not a good thing:<p>In an ideal world, you would <i>know</i> what encoding bytes are in and could therefore decode them explicitly using the known correct encoding, and this library would be redundant.<p>If instead, as is often the case in the real world, the coding is <i>unknown</i>, there exists the question of how to resolve the <i>numerous ambiguities</i> which result. A library such as this would have to <i>guess</i> what encoding to use in each specific instance, and the choices it ideally should make are <i>extremely</i> dependent on the circumstances and even the immediate context. As it is, the library is hard-coded with some specific algorithms to choose some encodings over others, and if those assumptions does not match your use case <i>exactly</i>, the library will corrupt your data.<p>A much better solution would perhaps involve a machine learning solution to the problem, and having the library be trained to deduce the probable encodings from a large set of example data from each user’s <i>individual</i> use case. Even these will occasionally be wrong, but at least it would be the best we could do with unknown encodings without resorting to manual processing.<p>However, a <i>one-size-fits-all</i> “solution” such as this is merely giving people a further excuse to keep not caring about encodings, to pretend that encodings can be “detected”, and that there exists such a thing as “plain text”.<p>[…]<p>I have […] two main arguments:<p>1. Due to its simplicity for a large group of naïve users, the library will likely be prone to over- and misuse. Since the library uses guessing as its method of decoding, and by definition a guess may be wrong, this will lead to some unnecessary data corruption in situations where use of this library (and the resulting data corruption) was not actually needed.<p>2. The library uses a one-size-fits-all model in the area of guessing encodings and language. This has historically proven to be less than a good idea, since different users in different situations use different data and encodings, and [the] library’s algorithm will not fit all situations equally well. I [suggest] that a more tunable and customizable approach would indeed be the best one could do in the cases where the encoding is actually not known. (This minor complexity in use of the library would also have the benefit of discouraging overuse in unwarranted situations, thus also resolving the first point, above.)