<p dir="auto">GET request to an API returned a response with broken encoding.<br>
More specifically, the response body was encoded in ASCII even though the data sent via API is encoded in UTF-8.<br>
The result is escaped non-ASCII characters in the response body.<br>
This behaviour can be observed on r.content, r.text, and r.json().</p>
<p dir="auto">Following the quickstart guide, I tried to "force encoding" with <code class="notranslate">r.encoding = 'UTF-8'</code> but it had no effect.<br>
I made sure to include <code class="notranslate">charset=UTF-8</code> in request header, which also didn't help.</p>
<p dir="auto">What ended up working was <code class="notranslate">json.loads(r.text)</code>.</p>
<p dir="auto">Very simple I know, but locating the problem was difficult,<br>
not to mention I had to be absolutely sure the API was sending a data that was encoded in UTF-8.<br>
This problem is not mentioned or, at least, not easily found on Stackoverflow.<br>
I thought I would mention it here, so that others can find it.</p>
<p dir="auto">issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="693330603" data-permission-text="Title is private" data-url="https://github.com/psf/requests/issues/5577" data-hovercard-type="issue" data-hovercard-url="/psf/requests/issues/5577/hovercard" href="https://github.com/psf/requests/issues/5577">#5577</a> seems to be similar in nature, which only proves that Encoding is an issue that seems to pop up every now and then.<br>
The answer given for that issue seems to be "there is an easy solution so there's no need to fix it on our ends".<br>
But nonetheless, a bug is a bug.<br>
Besides, it would be nice to mention how to troubleshoot encoding on the quickstart guide.</p>
<p dir="auto">Sorry it's not so much of a "summary", but things had to be said.</p>
<h2 dir="auto">Expected Result</h2>
<p dir="auto">A response body properly encoded in UTF-8.<br>
Like so, <code class="notranslate">{'body': '상품등록'}</code></p>
<h2 dir="auto">Actual Result</h2>
<p dir="auto">A response body encoded in ASCII even though it's supposed to be UTF-8.<br>
<code class="notranslate">{'body': '\uc0c1\ud488\ub4f1\ub85'}</code></p>
<h2 dir="auto">Reproduction Steps</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import requests

url = 'https://www.sorryicannottellyoutheactualapi.com/api'
headers = {'Content-Type': 'application/json;charset=UTF-8'}
r = requests.get(url, headers=headers)
print(r.content)
print(r.text)
print(r.json())
"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">requests</span>

<span class="pl-s1">url</span> <span class="pl-c1">=</span> <span class="pl-s">'https://www.sorryicannottellyoutheactualapi.com/api'</span>
<span class="pl-s1">headers</span> <span class="pl-c1">=</span> {<span class="pl-s">'Content-Type'</span>: <span class="pl-s">'application/json;charset=UTF-8'</span>}
<span class="pl-s1">r</span> <span class="pl-c1">=</span> <span class="pl-s1">requests</span>.<span class="pl-en">get</span>(<span class="pl-s1">url</span>, <span class="pl-s1">headers</span><span class="pl-c1">=</span><span class="pl-s1">headers</span>)
<span class="pl-en">print</span>(<span class="pl-s1">r</span>.<span class="pl-s1">content</span>)
<span class="pl-en">print</span>(<span class="pl-s1">r</span>.<span class="pl-s1">text</span>)
<span class="pl-en">print</span>(<span class="pl-s1">r</span>.<span class="pl-en">json</span>())</pre></div>
<h2 dir="auto">System Information</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="chardet 3.0.4
cryptography 2.8
idna 2.9
implementation CPython 3.6.7
platform Linux 4.15.0-101-generic
pyOpenSSL 19.1.0 (openssl 1010101f)
requests 2.23.0
system_ssl 1010104f
urllib3 1.25.7
using_pyopenssl True

python 3.6.7"><pre class="notranslate"><code class="notranslate">chardet 3.0.4
cryptography 2.8
idna 2.9
implementation CPython 3.6.7
platform Linux 4.15.0-101-generic
pyOpenSSL 19.1.0 (openssl 1010101f)
requests 2.23.0
system_ssl 1010104f
urllib3 1.25.7
using_pyopenssl True

python 3.6.7
</code></pre></div>
<p dir="auto">The location header in a 302 redirect response is not being parsed correctly when it has special characters in it.</p>
<p dir="auto"><code class="notranslate">/Login.aspx?ReturnUrl=%2f</code> becomes <code class="notranslate">/Login.aspx%3FReturnUrl%3D/</code></p>
<p dir="auto">When followed, this results in a 404 because it's not correct.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; import requests
&gt;&gt;&gt; r = requests.get('http://affiliatenetwork.ist-apps.com')
&gt;&gt;&gt; r.status_code
404
&gt;&gt;&gt; print r.headers
{'content-length': '1245', 'x-powered-by': 'ASP.NET', 'date': 'Tue, 18 Oct 2011 09:15:08 GMT', 'content-type': 'text/html', 'connection': 'close', 'server': 'Microsoft-IIS/7.0'}
&gt;&gt;&gt; r.url
'http://affiliatenetwork.ist-apps.com/Login.aspx%3FReturnUrl%3D/'
&gt;&gt;&gt; r.history
[&lt;Response [302]&gt;]
&gt;&gt;&gt; r.history[0].headers
{'content-length': '142', 'x-powered-by': 'ASP.NET', 'x-aspnet-version': '4.0.30319', 'server': 'Microsoft-IIS/7.0', 'connection': 'close', 'location': '/Login.aspx?ReturnUrl=%2f', 'cache-control': 'private', 'date': 'Tue, 18 Oct 2011 10:56:08 GMT', 'content-type': 'text/html; charset=utf-8'}"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">import</span> <span class="pl-s1">requests</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">r</span> <span class="pl-c1">=</span> <span class="pl-s1">requests</span>.<span class="pl-en">get</span>(<span class="pl-s">'http://affiliatenetwork.ist-apps.com'</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">r</span>.<span class="pl-s1">status_code</span>
<span class="pl-c1">404</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">print</span> <span class="pl-s1">r</span>.<span class="pl-s1">headers</span>
{<span class="pl-s">'content-length'</span>: <span class="pl-s">'1245'</span>, <span class="pl-s">'x-powered-by'</span>: <span class="pl-s">'ASP.NET'</span>, <span class="pl-s">'date'</span>: <span class="pl-s">'Tue, 18 Oct 2011 09:15:08 GMT'</span>, <span class="pl-s">'content-type'</span>: <span class="pl-s">'text/html'</span>, <span class="pl-s">'connection'</span>: <span class="pl-s">'close'</span>, <span class="pl-s">'server'</span>: <span class="pl-s">'Microsoft-IIS/7.0'</span>}
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">r</span>.<span class="pl-s1">url</span>
<span class="pl-s">'http://affiliatenetwork.ist-apps.com/Login.aspx%3FReturnUrl%3D/'</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">r</span>.<span class="pl-s1">history</span>
[<span class="pl-c1">&lt;</span><span class="pl-v">Response</span> [<span class="pl-c1">302</span>]<span class="pl-c1">&gt;</span>]
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">r</span>.<span class="pl-s1">history</span>[<span class="pl-c1">0</span>].<span class="pl-s1">headers</span>
{<span class="pl-s">'content-length'</span>: <span class="pl-s">'142'</span>, <span class="pl-s">'x-powered-by'</span>: <span class="pl-s">'ASP.NET'</span>, <span class="pl-s">'x-aspnet-version'</span>: <span class="pl-s">'4.0.30319'</span>, <span class="pl-s">'server'</span>: <span class="pl-s">'Microsoft-IIS/7.0'</span>, <span class="pl-s">'connection'</span>: <span class="pl-s">'close'</span>, <span class="pl-s">'location'</span>: <span class="pl-s">'/Login.aspx?ReturnUrl=%2f'</span>, <span class="pl-s">'cache-control'</span>: <span class="pl-s">'private'</span>, <span class="pl-s">'date'</span>: <span class="pl-s">'Tue, 18 Oct 2011 10:56:08 GMT'</span>, <span class="pl-s">'content-type'</span>: <span class="pl-s">'text/html; charset=utf-8'</span>}</pre></div>
<p dir="auto">curl parses this correctly:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="david@Shibuya ~: curl -v http://affiliatenetwork.ist-apps.com
* About to connect() to affiliatenetwork.ist-apps.com port 80 (#0)
*   Trying 94.236.92.60... connected
* Connected to affiliatenetwork.ist-apps.com (94.236.92.60) port 80 (#0)
&gt; GET / HTTP/1.1
&gt; User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
&gt; Host: affiliatenetwork.ist-apps.com
&gt; Accept: */*
&gt; 
&lt; HTTP/1.1 302 Found
&lt; Cache-Control: private
&lt; Content-Type: text/html; charset=utf-8
&lt; Location: /Login.aspx?ReturnUrl=%2f
&lt; Server: Microsoft-IIS/7.0
&lt; X-AspNet-Version: 4.0.30319
&lt; X-Powered-By: ASP.NET
&lt; Date: Tue, 18 Oct 2011 11:26:34 GMT
&lt; Content-Length: 142
&lt; 
&lt;html&gt;&lt;head&gt;&lt;title&gt;Object moved&lt;/title&gt;&lt;/head&gt;&lt;body&gt;
&lt;h2&gt;Object moved to &lt;a href=&quot;/Login.aspx?ReturnUrl=%2f&quot;&gt;here&lt;/a&gt;.&lt;/h2&gt;
&lt;/body&gt;&lt;/html&gt;
* Connection #0 to host affiliatenetwork.ist-apps.com left intact
* Closing connection #0"><pre class="notranslate">david@Shibuya <span class="pl-k">~</span>: curl -v http://affiliatenetwork.ist-apps.com
<span class="pl-k">*</span> About to <span class="pl-en">connect</span>() to affiliatenetwork.ist-apps.com port 80 (<span class="pl-c"><span class="pl-c">#</span>0)</span>
<span class="pl-k">*</span>   Trying 94.236.92.60... connected
<span class="pl-k">*</span> Connected to affiliatenetwork.ist-apps.com (94.236.92.60) port 80 (<span class="pl-c"><span class="pl-c">#</span>0)</span>
<span class="pl-k">&gt;</span> GET / HTTP/1.1
<span class="pl-k">&gt;</span> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
<span class="pl-k">&gt;</span> Host: affiliatenetwork.ist-apps.com
<span class="pl-k">&gt;</span> Accept: <span class="pl-k">*</span>/<span class="pl-k">*</span>
<span class="pl-k">&gt;</span> 
<span class="pl-k">&lt;</span> HTTP/1.1 302 Found
<span class="pl-k">&lt;</span> Cache-Control: private
<span class="pl-k">&lt;</span> Content-Type: text/html<span class="pl-k">;</span> charset=utf-8
<span class="pl-k">&lt;</span> Location: /Login.aspx<span class="pl-k">?</span>ReturnUrl=%2f
<span class="pl-k">&lt;</span> Server: Microsoft-IIS/7.0
<span class="pl-k">&lt;</span> X-AspNet-Version: 4.0.30319
<span class="pl-k">&lt;</span> X-Powered-By: ASP.NET
<span class="pl-k">&lt;</span> Date: Tue, 18 Oct 2011 11:26:34 GMT
<span class="pl-k">&lt;</span> Content-Length: 142
<span class="pl-k">&lt;</span> 
<span class="pl-k">&lt;</span>html&gt;&lt;head&gt;&lt;title<span class="pl-k">&gt;</span>Object moved<span class="pl-k">&lt;</span>/title&gt;&lt;/head&gt;&lt;body<span class="pl-k">&gt;</span>
<span class="pl-k">&lt;</span>h<span class="pl-k">2&gt;</span>Object moved to <span class="pl-k">&lt;</span>a href=<span class="pl-s"><span class="pl-pds">"</span>/Login.aspx?ReturnUrl=%2f<span class="pl-pds">"</span></span><span class="pl-k">&gt;</span>here<span class="pl-k">&lt;</span>/a<span class="pl-k">&gt;</span>.<span class="pl-k">&lt;</span>/h<span class="pl-k">2&gt;</span>
<span class="pl-k">&lt;</span>/body&gt;&lt;/html<span class="pl-k">&gt;</span>
<span class="pl-k">*</span> Connection <span class="pl-c"><span class="pl-c">#</span>0 to host affiliatenetwork.ist-apps.com left intact</span>
<span class="pl-k">*</span> Closing connection <span class="pl-c"><span class="pl-c">#</span>0</span></pre></div>
<p dir="auto">httplib also parses this correctly:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; conn = httplib.HTTPConnection('affiliatenetwork.ist-apps.com')
&gt;&gt;&gt; conn.request(&quot;GET&quot;,&quot;/&quot;)
&gt;&gt;&gt; res = conn.getresponse()
&gt;&gt;&gt; print res.status
302
&gt;&gt;&gt; print res.reason
Found
&gt;&gt;&gt; print res.getheaders()
[('content-length', '142'), ('x-powered-by', 'ASP.NET'), ('x-aspnet-version', '4.0.30319'), ('server', 'Microsoft-IIS/7.0'), ('location', '/Login.aspx?ReturnUrl=%2f'), ('cache-control', 'private'), ('date', 'Tue, 18 Oct 2011 10:35:20 GMT'), ('content-type', 'text/html; charset=utf-8')]"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">conn</span> <span class="pl-c1">=</span> <span class="pl-s1">httplib</span>.<span class="pl-v">HTTPConnection</span>(<span class="pl-s">'affiliatenetwork.ist-apps.com'</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">conn</span>.<span class="pl-en">request</span>(<span class="pl-s">"GET"</span>,<span class="pl-s">"/"</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">res</span> <span class="pl-c1">=</span> <span class="pl-s1">conn</span>.<span class="pl-en">getresponse</span>()
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">print</span> <span class="pl-s1">res</span>.<span class="pl-s1">status</span>
<span class="pl-c1">302</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">print</span> <span class="pl-s1">res</span>.<span class="pl-s1">reason</span>
<span class="pl-v">Found</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">print</span> <span class="pl-s1">res</span>.<span class="pl-en">getheaders</span>()
[(<span class="pl-s">'content-length'</span>, <span class="pl-s">'142'</span>), (<span class="pl-s">'x-powered-by'</span>, <span class="pl-s">'ASP.NET'</span>), (<span class="pl-s">'x-aspnet-version'</span>, <span class="pl-s">'4.0.30319'</span>), (<span class="pl-s">'server'</span>, <span class="pl-s">'Microsoft-IIS/7.0'</span>), (<span class="pl-s">'location'</span>, <span class="pl-s">'/Login.aspx?ReturnUrl=%2f'</span>), (<span class="pl-s">'cache-control'</span>, <span class="pl-s">'private'</span>), (<span class="pl-s">'date'</span>, <span class="pl-s">'Tue, 18 Oct 2011 10:35:20 GMT'</span>), (<span class="pl-s">'content-type'</span>, <span class="pl-s">'text/html; charset=utf-8'</span>)]</pre></div>
<p dir="auto">Looks like the problem is caused by <a href="https://github.com/kennethreitz/requests/blob/master/requests/models.py#L237">https://github.com/kennethreitz/requests/blob/master/requests/models.py#L237</a></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; url = '/Login.aspx?ReturnUrl=%2f'
&gt;&gt;&gt; o = urlparse.urlparse(url)
&gt;&gt;&gt; print o
ParseResult(scheme='', netloc='', path='/Login.aspx', params='', query='ReturnUrl=%2f', fragment='')
&gt;&gt;&gt; import urllib
&gt;&gt;&gt; urllib.quote(urllib.unquote(url))
'/Login.aspx%3FReturnUrl%3D/'
&gt;&gt;&gt; urllib.unquote(url)
'/Login.aspx?ReturnUrl=/'"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">url</span> <span class="pl-c1">=</span> <span class="pl-s">'/Login.aspx?ReturnUrl=%2f'</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">o</span> <span class="pl-c1">=</span> <span class="pl-s1">urlparse</span>.<span class="pl-en">urlparse</span>(<span class="pl-s1">url</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">print</span> <span class="pl-s1">o</span>
<span class="pl-v">ParseResult</span>(<span class="pl-s1">scheme</span><span class="pl-c1">=</span><span class="pl-s">''</span>, <span class="pl-s1">netloc</span><span class="pl-c1">=</span><span class="pl-s">''</span>, <span class="pl-s1">path</span><span class="pl-c1">=</span><span class="pl-s">'/Login.aspx'</span>, <span class="pl-s1">params</span><span class="pl-c1">=</span><span class="pl-s">''</span>, <span class="pl-s1">query</span><span class="pl-c1">=</span><span class="pl-s">'ReturnUrl=%2f'</span>, <span class="pl-s1">fragment</span><span class="pl-c1">=</span><span class="pl-s">''</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">import</span> <span class="pl-s1">urllib</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">urllib</span>.<span class="pl-en">quote</span>(<span class="pl-s1">urllib</span>.<span class="pl-en">unquote</span>(<span class="pl-s1">url</span>))
<span class="pl-s">'/Login.aspx%3FReturnUrl%3D/'</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">urllib</span>.<span class="pl-en">unquote</span>(<span class="pl-s1">url</span>)
<span class="pl-s">'/Login.aspx?ReturnUrl=/'</span></pre></div>