title: random text from /dev/random raising an error in lxml: All strings must be XML compatible: Unicode or ASCII, no NULL bytes
body: <p>I am, for the sake of testing my web app, pasting some random characters from /dev/random into my web frontend. This line throws an error:</p>
  
  <pre><code>print repr(comment)
  import html5lib
  print html5lib.parse(comment, treebuilder="lxml")
  
  'a\xef\xbf\xbd\xef\xbf\xbd\xc9\xb6E\xef\xbf\xbd\xef\xbf\xbd`\xef\xbf\xbd]\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd2 \x14\xef\xbf\xbd\xc7\xbe\xef\xbf\xbdy\xcb\x9c\xef\xbf\xbdi1O\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbdZ\xef\xbf\xbd.\xef\xbf\xbd\x17^C'
  
  Unhandled Error
      Traceback (most recent call last):
        File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 893, in _inlineCallbacks
          result = g.send(result)
        File "/home/work/random/social/social/item.py", line 389, in _new
          convId, conv = yield plugin.create(request)
        File "/home/work/random/social/social/logging.py", line 47, in wrapper
          ret = func(*args, **kwargs)
        File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 1014, in unwindGenerator
          return _inlineCallbacks(None, f(*args, **kwargs), Deferred())
      --- &lt;exception caught here&gt; ---
        File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 893, in _inlineCallbacks
          result = g.send(result)
        File "/home/work/random/social/twisted/plugins/status.py", line 63, in create
          print html5lib.parse(comment, treebuilder="lxml")
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/html5parser.py", line 38, in parse
          return p.parse(doc, encoding=encoding)
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/html5parser.py", line 211, in parse
          parseMeta=parseMeta, useChardet=useChardet)
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/html5parser.py", line 111, in _parse
          self.mainLoop()
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/html5parser.py", line 174, in mainLoop
          self.phase.processCharacters(token)
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/html5parser.py", line 572, in processCharacters
          self.parser.phase.processCharacters(token)
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/html5parser.py", line 611, in processCharacters
          self.parser.phase.processCharacters(token)
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/html5parser.py", line 652, in processCharacters
          self.parser.phase.processCharacters(token)
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/html5parser.py", line 711, in processCharacters
          self.parser.phase.processCharacters(token)
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/html5parser.py", line 804, in processCharacters
          self.parser.phase.processCharacters(token)
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/html5parser.py", line 948, in processCharacters
          self.tree.insertText(token["data"])
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/treebuilders/_base.py", line 288, in insertText
          parent.insertText(data)
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/treebuilders/etree_lxml.py", line 225, in insertText
          builder.Element.insertText(self, data, insertBefore)
        File "/usr/local/lib/python2.6/dist-packages/html5lib-0.90-py2.6.egg/html5lib/treebuilders/etree.py", line 114, in insertText
          self._element.text += data
        File "lxml.etree.pyx", line 821, in lxml.etree._Element.text.__set__ (src/lxml/lxml.etree.c:33308)
  
        File "apihelpers.pxi", line 646, in lxml.etree._setNodeText (src/lxml/lxml.etree.c:15287)
  
        File "apihelpers.pxi", line 1295, in lxml.etree._utf8 (src/lxml/lxml.etree.c:20212)
  
      exceptions.ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes
  </code></pre>
  
  <p>Before I am committing a user entered string, I am doing this:</p>
  
  <p>comment.decode('utf-8').encode('utf-8', "replace")</p>
  
  <p>but this does not seem to be helping in this case.</p>
  
  <p>--
  Abhi</p>