python requests.get() returns improperly decoded text instead of UTF-8?
The default assumed content encoding for text/html is ISO-8859-1 aka Latin-1 :( See RFC-2854. UTF-8 was too young to become the default, it was born in 1993, about the same time as HTML and HTTP.
Use .content to access the byte stream, or .text to access the decoded Unicode stream. If the HTTP server does not care about the correct encoding, the value of .text may be off.
When the content-type of the server is 'Content-Type:text/html', requests.get() returns improperly encoded data.
However, if we have the content type explicitly as 'Content-Type:text/html; charset=utf-8', it returns properly encoded data.
Also, when we use urllib.urlopen(), it returns properly encoded data. 
Has anyone noticed this before? Why does requests.get() behave like this?