<p dir="auto">Hi,</p>
<p dir="auto">I'm pretty desperately trying to nail down the reason that an url unshortener script is freezing from time to time.</p>
<p dir="auto">I'm using requests with several head and get-requests, such as</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="with requests.Session as s:
    r = s.get(url, allow_redirects=True, timeout=self.timeout, headers=self.headers, verify=False)"><pre class="notranslate"><code class="notranslate">with requests.Session as s:
    r = s.get(url, allow_redirects=True, timeout=self.timeout, headers=self.headers, verify=False)
</code></pre></div>
<p dir="auto">where self.timeout are 10 seconds and the headers just include a user-agent.</p>
<p dir="auto">The request is called by a worker in a multiprocessing Pool with 10 workers that get's constantly feeded via Pool.imap() with a list of urls.</p>
<p dir="auto">When keyboard interrupting I get from some of the workers the following Traceback:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;---/dataprocessor.py&quot;, line 98, in try_unshorten_expanded_urls
    result = unshorten_expanded_urls(tweet_dataframe_tuple)
  File &quot;---/dataprocessor.py&quot;, line 84, in unshorten_expanded_urls
    'unshortened_url': u.unshorten(expanded_urls[i]),
  File &quot;---/unshorten.py&quot;, line 162, in unshorten
    verify=False)
  File &quot;/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/sessions.py&quot;, line 480, in get
    return self.request('GET', url, **kwargs)
  File &quot;/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/sessions.py&quot;, line 468, in request
    resp = self.send(prep, **send_kwargs)
  File &quot;/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/sessions.py&quot;, line 608, in send
    r.content
  File &quot;/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/models.py&quot;, line 737, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File &quot;/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/models.py&quot;, line 660, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File &quot;/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/packages/urllib3/response.py&quot;, line 344, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File &quot;/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/packages/urllib3/response.py&quot;, line 301, in read
    data = self._fp.read(amt)
  File &quot;/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/http/client.py&quot;, line 433, in read
    n = self.readinto(b)
  File &quot;/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/http/client.py&quot;, line 473, in readinto
    n = self.fp.readinto(b)
  File &quot;/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/socket.py&quot;, line 575, in readinto
    return self._sock.recv_into(b)
KeyboardInterrupt"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "---/dataprocessor.py", line 98, in try_unshorten_expanded_urls
    result = unshorten_expanded_urls(tweet_dataframe_tuple)
  File "---/dataprocessor.py", line 84, in unshorten_expanded_urls
    'unshortened_url': u.unshorten(expanded_urls[i]),
  File "---/unshorten.py", line 162, in unshorten
    verify=False)
  File "/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/sessions.py", line 480, in get
    return self.request('GET', url, **kwargs)
  File "/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/sessions.py", line 608, in send
    r.content
  File "/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/models.py", line 737, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/models.py", line 660, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 344, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/http/client.py", line 433, in read
    n = self.readinto(b)
  File "/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/http/client.py", line 473, in readinto
    n = self.fp.readinto(b)
  File "/home/ubuntu/miniconda3/envs/TQ/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
KeyboardInterrupt
</code></pre></div>
<p dir="auto">If the function would raise an exception, I've made sure that it would be catched within the function so that it would not block the pool. But as it seems, <code class="notranslate">readinto</code> is just hanging here (for hours). I don't know whether this is related to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="143481469" data-permission-text="Title is private" data-url="https://github.com/psf/requests/issues/3066" data-hovercard-type="issue" data-hovercard-url="/psf/requests/issues/3066/hovercard" href="https://github.com/psf/requests/issues/3066">#3066</a>, as I am not familiar enough with the protocols or requests itself, so please excuse a duplicate if this should be one or if the error is my bad.</p>
<p dir="auto">The freeze happened several times with the same batch of urls now.<br>
I tried to find the exact url causing this freeze but did not succeed as the batch contains several thousand, though I've already excluded everything that exceeds a certain content length or does not have text as content type via a head request before the get.</p>
<p dir="auto">I am running requests 2.10.0 with Python 3.5.1 in a miniconda environment installed via pip on an Ubuntu 14.04.</p>
<p dir="auto">However, great library, hope this helps to make it better.</p>
<p dir="auto">Cheers!</p>
<p dir="auto">UPDATE:</p>
<p dir="auto">Here the relevant code for the traceback (line 162 in unshorten.py):</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def check_size_and_type(request_object, max_content_length):
    try:
        length = int(request_object.headers['content-length'])
    except:
        length = None
    try:
        type = str(request_object.headers['content-type'])
    except:
        type = ''
    if ((length is None or length &lt; max_content_length) and
        (type.startswith('text/html') or
         type.startswith('application/xhtml') or
         type.startswith('text/xml'))):
        return True
    else:
        return False

with requests.Session() as s:
    r = s.head(url, allow_redirects=True,
                     timeout=self.timeout,
                     headers=self.headers,
                     verify=False)

if check_size_and_type(r, self.max_content_length):
    r = s.get(url, allow_redirects=True,
    timeout=self.timeout,
    headers=self.headers,
    verify=False)"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">check_size_and_type</span>(<span class="pl-s1">request_object</span>, <span class="pl-s1">max_content_length</span>):
    <span class="pl-k">try</span>:
        <span class="pl-s1">length</span> <span class="pl-c1">=</span> <span class="pl-en">int</span>(<span class="pl-s1">request_object</span>.<span class="pl-s1">headers</span>[<span class="pl-s">'content-length'</span>])
    <span class="pl-k">except</span>:
        <span class="pl-s1">length</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>
    <span class="pl-k">try</span>:
        <span class="pl-s1">type</span> <span class="pl-c1">=</span> <span class="pl-en">str</span>(<span class="pl-s1">request_object</span>.<span class="pl-s1">headers</span>[<span class="pl-s">'content-type'</span>])
    <span class="pl-k">except</span>:
        <span class="pl-s1">type</span> <span class="pl-c1">=</span> <span class="pl-s">''</span>
    <span class="pl-k">if</span> ((<span class="pl-s1">length</span> <span class="pl-c1">is</span> <span class="pl-c1">None</span> <span class="pl-c1">or</span> <span class="pl-s1">length</span> <span class="pl-c1">&lt;</span> <span class="pl-s1">max_content_length</span>) <span class="pl-c1">and</span>
        (<span class="pl-s1">type</span>.<span class="pl-en">startswith</span>(<span class="pl-s">'text/html'</span>) <span class="pl-c1">or</span>
         <span class="pl-s1">type</span>.<span class="pl-en">startswith</span>(<span class="pl-s">'application/xhtml'</span>) <span class="pl-c1">or</span>
         <span class="pl-s1">type</span>.<span class="pl-en">startswith</span>(<span class="pl-s">'text/xml'</span>))):
        <span class="pl-k">return</span> <span class="pl-c1">True</span>
    <span class="pl-k">else</span>:
        <span class="pl-k">return</span> <span class="pl-c1">False</span>

<span class="pl-k">with</span> <span class="pl-s1">requests</span>.<span class="pl-v">Session</span>() <span class="pl-k">as</span> <span class="pl-s1">s</span>:
    <span class="pl-s1">r</span> <span class="pl-c1">=</span> <span class="pl-s1">s</span>.<span class="pl-en">head</span>(<span class="pl-s1">url</span>, <span class="pl-s1">allow_redirects</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
                     <span class="pl-s1">timeout</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">timeout</span>,
                     <span class="pl-s1">headers</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">headers</span>,
                     <span class="pl-s1">verify</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)

<span class="pl-k">if</span> <span class="pl-en">check_size_and_type</span>(<span class="pl-s1">r</span>, <span class="pl-s1">self</span>.<span class="pl-s1">max_content_length</span>):
    <span class="pl-s1">r</span> <span class="pl-c1">=</span> <span class="pl-s1">s</span>.<span class="pl-en">get</span>(<span class="pl-s1">url</span>, <span class="pl-s1">allow_redirects</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
    <span class="pl-s1">timeout</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">timeout</span>,
    <span class="pl-s1">headers</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">headers</span>,
    <span class="pl-s1">verify</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)</pre></div>
<p dir="auto">Does <code class="notranslate">requests</code> currently support Multipart-encoded files in <code class="notranslate">PUT</code> requests?</p>
<p dir="auto">It seems that both<br>
<a href="http://docs.python-requests.org/en/master/user/quickstart/#post-a-multipart-encoded-file" rel="nofollow">http://docs.python-requests.org/en/master/user/quickstart/#post-a-multipart-encoded-file</a><br>
and<br>
<a href="https://toolbelt.readthedocs.io/en/latest/uploading-data.html" rel="nofollow">https://toolbelt.readthedocs.io/en/latest/uploading-data.html</a><br>
only work with <code class="notranslate">POST</code>. For <code class="notranslate">PUT</code> requests, I keep getting error "413 Request Entity Too Large" from my Apache httpd.</p>
<p dir="auto">Should that be considered a bug, improvement suggestion, or there is some fundamental limitation why <code class="notranslate">PUT</code> shouldn't upload large files? (yes, my <code class="notranslate">GET</code> would stream the huge file unmodified)</p>
<p dir="auto">Original StackOverflow question:<br>
<a href="https://stackoverflow.com/questions/50533901/multipart-put-with-requests" rel="nofollow">https://stackoverflow.com/questions/50533901/multipart-put-with-requests</a></p>