Convert XML/HTML Entities into Unicode String in Python
<p>I'm doing some web scraping and sites frequently use HTML entities to represent non ascii characters. Does Python have a utility that takes a string with HTML entities and returns a unicode type?</p>

<p>For example:</p>

<p>I get back:</p>

<pre><code>&amp;#x01ce;
</code></pre>

<p>which represents an "Ç" with a tone mark.  In binary, this is represented as the 16 bit 01ce.  I want to convert the html entity into the value  <code>u'\u01ce'</code></p>

<p>Python has the <a href="https://docs.python.org/2/library/htmllib.html#module-htmlentitydefs">htmlentitydefs</a> module, but this doesn't include a function to unescape HTML entities.</p>

<p>Python developer Fredrik Lundh (author of elementtree, among other things) has  such a function <a href="http://effbot.org/zone/re-sub.htm#unescape-html">on his website</a>, which works with decimal, hex and named entities:</p>

<pre><code>import re, htmlentitydefs

##
# Removes HTML or XML character references and entities from a text string.
#
# @param text The HTML (or XML) source text.
# @return The plain text, as a Unicode string, if necessary.

def unescape(text):
    def fixup(m):
        text = m.group(0)
        if text[:2] == "&amp;#":
            # character reference
            try:
                if text[:3] == "&amp;#x":
                    return unichr(int(text[3:-1], 16))
                else:
                    return unichr(int(text[2:-1]))
            except ValueError:
                pass
        else:
            # named entity
            try:
                text = unichr(htmlentitydefs.name2codepoint[text[1:-1]])
            except KeyError:
                pass
        return text # leave as is
    return re.sub("&amp;#?\w+;", fixup, text)
</code></pre>
