title: Python, convert HTML entities to Unicode
body: <p>(Edit: I'm using Python 2.7)
  (Edit 2: I have already checked <a href="https://stackoverflow.com/questions/57708/convert-xml-html-entities-into-unicode-string-in-python">Convert XML/HTML Entities into Unicode String in Python</a>, the solutions do not work. Please do not flag this as already answered.)</p>
  
  <p>I've been unable to find a python package that can reliably convert text with some html entities in it.  I've found that HTMLParser works for some stuff but also breaks a lot.  BeautifulSoup never seems to work for converting to unicode.  How can I return a unicode representation of strings a-d using only one method?  </p>
  
  <p>I think the problem I'm having is that some of my text has both unicode characters and html entities (as in example string d).</p>
  
  <pre><code>import HTMLParser
  from bs4 import BeautifulSoup
  
  astring = "P&amp;amp;O."
  bstring = "&amp;amp; "
  cstring = "&amp;gt;"
  dstring = "&amp;gt; 150ÎC"
  
  pars = HTMLParser.HTMLParser()
  a1 = BeautifulSoup(astring)
  a2 = pars.unescape(astring)
  print "a1:", a1
  print "a2:", a2
  b1 = BeautifulSoup(bstring)
  b2 = pars.unescape(bstring)
  print "b1:", b1
  print "b2:", b2
  c1 = BeautifulSoup(cstring)
  c2 = pars.unescape(cstring)
  print "c1:", c1
  print "c2:", c2
  d1 = BeautifulSoup(dstring)
  try: d2 = pars.unescape(dstring)
  except:d2 = "HTML Parse Broke!"
  print "d1:", d1
  print "d2:", d2
  </code></pre>
  
  <p>which gives the following output:</p>
  
  <pre><code>a1: P&amp;amp;O.
  a2: P&amp;O.
  b1: &amp;amp; 
  b2: &amp; 
  c1: &amp;gt;
  c2: &gt;
  d1: &amp;gt; 150ÎC
  d2: HTML Parse Broke!
  </code></pre>
  
  <p>Edit 3:  kalhartt's suggestion lead me to a solution.  To keep the strings with mixed character encoding from breaking I used .decode('utf-8')</p>