<p dir="auto">I'm trying to request google search results and access the top few links. However, it always gets stuck on some sites. Ctrl-C is not in effect but Ctrl-Z does bring me to the terminal.</p>
<p dir="auto">I have set a timeout for each request.<br>
I have read messages that DNS is one of the sources of blocking and I patched this with an HTTP adapter.<br>
It seems that the requests library is using blocking TCP connection(not sure), where the program has no control over the timeout if the underlying implementation does not return.</p>
<p dir="auto">In another scenario, the connection is perfectly slow or the page is unexpected too big such that this connection keeps forever. This case is obviously not a bug but a vulnerability that can be used to anti-spider.</p>
<p dir="auto">I would like to set a hard limit for my connection, regardless of DNS, TCP handshaking time, TLS handshaking time, or uploading/downloading time, and at any time I would like to exit, I just enter Ctrl-C. This can be done by using requests in another thread(POSIX thread)/process, ungracefully.</p>
<p dir="auto">My program just stuck again.</p>
<p dir="auto">The request <code class="notranslate">my_session.head(some_url, verify=True)</code> reuses existing connections, including any that were established with <code class="notranslate">verify=False</code>.</p>
<h2 dir="auto">Expected Result</h2>
<p dir="auto">I would expect that HTTP requests with <code class="notranslate">verify=True</code> would only reuse existing connections that were also established with <code class="notranslate">verify=True</code>.</p>
<h2 dir="auto">Actual Result</h2>
<p dir="auto">From my test-case, it seems that the session will reuse an existing connection, irrespective of how that connection was established.</p>
<h2 dir="auto">Reproduction Steps</h2>
<p dir="auto">The following code should demonstrate the problem.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import requests
from urllib3.exceptions import (MaxRetryError, SSLError)

def do_head_request_on(strategy, verify):
    try:
        r = strategy.head(&quot;https://wrong.host.badssl.com/&quot;,
                          verify=verify)
        print(&quot;With verify %s,\tHTTP request made OK.&quot; % verify)
    except requests.exceptions.ConnectionError as ce:
        cause = ce.__context__
        if type(cause) is MaxRetryError:
            cause = cause.reason
        if type(cause) is SSLError:    
            print(&quot;With verify %s,\tTLS problem&quot; % verify)
        else:
            raise ce

requests.packages.urllib3.disable_warnings()

print(&quot;NOT using sessions:&quot;)
do_head_request_on(requests, True)
do_head_request_on(requests, False)
do_head_request_on(requests, True)

session = requests.Session()

print(&quot;\nUSING sessions:&quot;)
do_head_request_on(session, True)
do_head_request_on(session, False)
do_head_request_on(session, True)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">requests</span>
<span class="pl-k">from</span> <span class="pl-s1">urllib3</span>.<span class="pl-s1">exceptions</span> <span class="pl-k">import</span> (<span class="pl-v">MaxRetryError</span>, <span class="pl-v">SSLError</span>)

<span class="pl-k">def</span> <span class="pl-en">do_head_request_on</span>(<span class="pl-s1">strategy</span>, <span class="pl-s1">verify</span>):
    <span class="pl-k">try</span>:
        <span class="pl-s1">r</span> <span class="pl-c1">=</span> <span class="pl-s1">strategy</span>.<span class="pl-en">head</span>(<span class="pl-s">"https://wrong.host.badssl.com/"</span>,
                          <span class="pl-s1">verify</span><span class="pl-c1">=</span><span class="pl-s1">verify</span>)
        <span class="pl-en">print</span>(<span class="pl-s">"With verify %s,<span class="pl-cce">\t</span>HTTP request made OK."</span> <span class="pl-c1">%</span> <span class="pl-s1">verify</span>)
    <span class="pl-k">except</span> <span class="pl-s1">requests</span>.<span class="pl-s1">exceptions</span>.<span class="pl-v">ConnectionError</span> <span class="pl-k">as</span> <span class="pl-s1">ce</span>:
        <span class="pl-s1">cause</span> <span class="pl-c1">=</span> <span class="pl-s1">ce</span>.<span class="pl-s1">__context__</span>
        <span class="pl-k">if</span> <span class="pl-en">type</span>(<span class="pl-s1">cause</span>) <span class="pl-c1">is</span> <span class="pl-v">MaxRetryError</span>:
            <span class="pl-s1">cause</span> <span class="pl-c1">=</span> <span class="pl-s1">cause</span>.<span class="pl-s1">reason</span>
        <span class="pl-k">if</span> <span class="pl-en">type</span>(<span class="pl-s1">cause</span>) <span class="pl-c1">is</span> <span class="pl-v">SSLError</span>:    
            <span class="pl-en">print</span>(<span class="pl-s">"With verify %s,<span class="pl-cce">\t</span>TLS problem"</span> <span class="pl-c1">%</span> <span class="pl-s1">verify</span>)
        <span class="pl-k">else</span>:
            <span class="pl-k">raise</span> <span class="pl-s1">ce</span>

<span class="pl-s1">requests</span>.<span class="pl-s1">packages</span>.<span class="pl-s1">urllib3</span>.<span class="pl-en">disable_warnings</span>()

<span class="pl-en">print</span>(<span class="pl-s">"NOT using sessions:"</span>)
<span class="pl-en">do_head_request_on</span>(<span class="pl-s1">requests</span>, <span class="pl-c1">True</span>)
<span class="pl-en">do_head_request_on</span>(<span class="pl-s1">requests</span>, <span class="pl-c1">False</span>)
<span class="pl-en">do_head_request_on</span>(<span class="pl-s1">requests</span>, <span class="pl-c1">True</span>)

<span class="pl-s1">session</span> <span class="pl-c1">=</span> <span class="pl-s1">requests</span>.<span class="pl-v">Session</span>()

<span class="pl-en">print</span>(<span class="pl-s">"<span class="pl-cce">\n</span>USING sessions:"</span>)
<span class="pl-en">do_head_request_on</span>(<span class="pl-s1">session</span>, <span class="pl-c1">True</span>)
<span class="pl-en">do_head_request_on</span>(<span class="pl-s1">session</span>, <span class="pl-c1">False</span>)
<span class="pl-en">do_head_request_on</span>(<span class="pl-s1">session</span>, <span class="pl-c1">True</span>)</pre></div>
<p dir="auto">Here is the output on my machine:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NOT using sessions:
With verify True,       TLS problem
With verify False,      HTTP request made OK.
With verify True,       TLS problem

USING sessions:
With verify True,       TLS problem
With verify False,      HTTP request made OK.
With verify True,       HTTP request made OK."><pre class="notranslate"><code class="notranslate">NOT using sessions:
With verify True,       TLS problem
With verify False,      HTTP request made OK.
With verify True,       TLS problem

USING sessions:
With verify True,       TLS problem
With verify False,      HTTP request made OK.
With verify True,       HTTP request made OK.
</code></pre></div>
<p dir="auto">The issue is that the final request has <code class="notranslate">verify=True</code> against a known-bad endpoint; however, the request is successful.</p>
<h2 dir="auto">System Information</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ python -m requests.help"><pre class="notranslate"><code class="notranslate">$ python -m requests.help
</code></pre></div>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;chardet&quot;: {
    &quot;version&quot;: &quot;4.0.0&quot;
  },
  &quot;cryptography&quot;: {
    &quot;version&quot;: &quot;37.0.4&quot;
  },
  &quot;idna&quot;: {
    &quot;version&quot;: &quot;2.10&quot;
  },
  &quot;implementation&quot;: {
    &quot;name&quot;: &quot;CPython&quot;,
    &quot;version&quot;: &quot;3.9.2&quot;
  },
  &quot;platform&quot;: {
    &quot;release&quot;: &quot;5.10.0-18-amd64&quot;,
    &quot;system&quot;: &quot;Linux&quot;
  },
  &quot;pyOpenSSL&quot;: {
    &quot;openssl_version&quot;: &quot;30000050&quot;,
    &quot;version&quot;: &quot;22.0.0&quot;
  },
  &quot;requests&quot;: {
    &quot;version&quot;: &quot;2.25.1&quot;
  },
  &quot;system_ssl&quot;: {
    &quot;version&quot;: &quot;101010ef&quot;
  },
  &quot;urllib3&quot;: {
    &quot;version&quot;: &quot;1.26.5&quot;
  },
  &quot;using_pyopenssl&quot;: true
}"><pre class="notranslate">{
  <span class="pl-ent">"chardet"</span>: {
    <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>4.0.0<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"cryptography"</span>: {
    <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>37.0.4<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"idna"</span>: {
    <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>2.10<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"implementation"</span>: {
    <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>CPython<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>3.9.2<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"platform"</span>: {
    <span class="pl-ent">"release"</span>: <span class="pl-s"><span class="pl-pds">"</span>5.10.0-18-amd64<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"system"</span>: <span class="pl-s"><span class="pl-pds">"</span>Linux<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"pyOpenSSL"</span>: {
    <span class="pl-ent">"openssl_version"</span>: <span class="pl-s"><span class="pl-pds">"</span>30000050<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>22.0.0<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"requests"</span>: {
    <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>2.25.1<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"system_ssl"</span>: {
    <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>101010ef<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"urllib3"</span>: {
    <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>1.26.5<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"using_pyopenssl"</span>: <span class="pl-c1">true</span>
}</pre></div>