Created on 2008-05-23 03:22 by vvro, last changed 2008-05-23 08:32 by lemburg. This issue is now closed.
The hashing algorithms don't support Unicode. Any Unicode text given to
them is first tried to convert ascii and then hashed. Not all strings
are convertible to ascii.
Now that Unicode is becoming the default encoding, specially for the web
side of python, where a lot of this hashing algorithms are used.
There should be some kind of Unicode support in them.
Example:
from hashlib import md5
md5(u'joão')
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe3' in
position 2: ordinal not in range(128).
Mark, do you concur?
You could just make a check for unicode strings and issue the encode in
the hash function.
I understand the byte abstraction, but if you issue an encode on a
unicode string with only ascii chars it gets converted to the same in
ascii, result will be the same.
So i got to do md5(u'joão'.encode("utf-8"))?
Wasn't unicode becoming the default?
If I do md5(u'john'), it works. And that's a unicode string. It should
have told me, no unicode then...
Only 2.6 should be marked. This is a feature request for an implicit
conversion with a default encoding; it is not a bugfix.
FWIW, here's a reference to an earlier discussion:
Also, it is unpersuasive that md5() works with u'john'. That is just
an artifact of the 2.x series. In 3.0, there is a more clear
distinction between bytes and text.
Recommending that this be rejected and closed. Without a universally
accepted (not just with Python) implicit coding, there's no way to
check MD5 checksums match for the same unicode text.
I'm rejecting this idea, for the reasons already given by others: the
same string might have different hash values, depending on which
encoding is chosen. Users will have to be explicit when hashing, just as
they need to be explicit when they chose a hash algorithm (i.e. md5,
sha1, or sha256 - they all do the same thing, but still produce
different output).
If you want a hash algorithm that abstracts from these details, use the
builtin hash function:
py> hash(u'joão')
679553179
On 2008-05-23 05:38, Raymond Hettinger wrote:
> Raymond Hettinger <rhettinger@users.sourceforge.net> added the comment:
>
>.
>
> Marc, do you concur?
Yes.
While we could fix an encoding to use for converting Unicode to
bytes, e.g. UTF-8, you clearly want hash functions to be portable
across platforms, programming languages and implementations.
Other languages or implementations might choose UTF-16 or some
other encoding, so it's not clear which encoding to choose and
there doesn't seem to be a standard for this either.
-1 on the idea. Martin already closed and rejected the idea for me.
Thanks,
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, May