title: Python sys.stdin throws a UnicodeDecodeError
body: <p>I'm trying to write a (very) basic web crawler using cURL and Python's <code>BeautifulSoup</code> library (since this is much easier to understand than GNU awk and a mess of regular expressions).</p>
  
  <p>Currently, I'm trying to pipe the contents of a webpage to the program with cURL (i.e., <code>curl http://www.example.com/ | ./parse-html.py</code>)</p>
  
  <p>For some reason, Python throws a <code>UnicodeDecodeError</code> because of an invalid start byte (I've looked at <a href="https://stackoverflow.com/questions/22216076/unicodedecodeerror-utf8-codec-cant-decode-byte-0xa5-in-position-0-invalid-s">this answer</a> and <a href="https://stackoverflow.com/questions/29095360/unicodedecodeerror-utf8-codec-cant-decode-byte-0x80-in-position-11-invalid">this answer</a> about invalid start bytes, but did not figure out how to solve the issue from them).</p>
  
  <p>Specifically, I've tried to use <code>a.encode('utf-8').split()</code> from the first answer. The second answer simply explained the issue (that Python found an invalid starter byte), though it didn't give a solution.</p>
  
  <p>I've attempted redirecting the output of cURL to a file (i.e., <code>curl http://www.example.com/ &gt; foobar.html</code> and modifying the program to accept a file as a command-line argument, though this causes the same <code>UnicodeDecodeError</code>.</p>
  
  <p>I've checked, and the output of <code>locale charmap</code> is <code>UTF-8</code>, which as far as I know, means that my system is encoding characters in <code>UTF-8</code> (which makes me especially confused about this <code>UnicodeDecodeError</code>.</p>
  
  <p>At the moment, the exact line causing the error is <code>html_doc = sys.stdin.readlines().encode('utf-8').strip()</code>. I've tried rewriting this as a for-loop, though I get the same issue.</p>
  
  <p>What exactly is causing the <code>UnicodeDecodeError</code> and how should I fix the issue?</p>
  
  <p><strong>EDIT:</strong>
  By changing the line <code>html_doc = sys.stdin.readlines().encode('utf-8').strip()</code> to <code>html_doc = sys.stdin</code> fixes the issue</p>