<p dir="auto">This issue is related to character encoding of the request parameters when sending a GET request. In my scenario, I need the percent character to not be encoded as %25. The <code class="notranslate">urllib.parse.urlencode</code> function has the <code class="notranslate">safe</code> input for this purpose. However, the <code class="notranslate">requests.get</code> function does not have such input. Therefore, I use <code class="notranslate">urllib.parse.urlencode</code> with <code class="notranslate">safe='%'</code> to get an ampersand-separated string of my request parameters and send that in for the <code class="notranslate">params</code> value of <code class="notranslate">requests.get</code>. Even with this trick, the response that I get is different than what I get with <code class="notranslate">urllib.request.Request</code>. The interesting thing is that when I print <code class="notranslate">response.url</code> (where <code class="notranslate">response</code> is the response object of <code class="notranslate">requests.get</code>), I see that the percent character is not encoded, but still the response is incorrect. It seems like encoding takes place somewhere else. I really hate abandoning the requests package because of this defect. I was wondering if there is any way this can be fixed in a future release.</p>
<h2 dir="auto">Expected Result</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;result&gt;
    &lt;TotalMatches&gt;1135&lt;/TotalMatches&gt;
    &lt;TotalPages&gt;57&lt;/TotalPages&gt;
    &lt;PageNumber&gt;1&lt;/PageNumber&gt;
   &lt;item&gt;
       &lt;mid&gt;36145&lt;/mid&gt;
       &lt;merchantname&gt; ..."><pre class="notranslate"><code class="notranslate">&lt;result&gt;
    &lt;TotalMatches&gt;1135&lt;/TotalMatches&gt;
    &lt;TotalPages&gt;57&lt;/TotalPages&gt;
    &lt;PageNumber&gt;1&lt;/PageNumber&gt;
   &lt;item&gt;
       &lt;mid&gt;36145&lt;/mid&gt;
       &lt;merchantname&gt; ...
</code></pre></div>
<h2 dir="auto">Actual Result</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;result&gt;
    &lt;TotalMatches&gt;0&lt;/TotalMatches&gt;
    &lt;TotalPages&gt;0&lt;/TotalPages&gt;
    &lt;PageNumber&gt;1&lt;/PageNumber&gt;
&lt;/result&gt; "><pre class="notranslate"><code class="notranslate">&lt;result&gt;
    &lt;TotalMatches&gt;0&lt;/TotalMatches&gt;
    &lt;TotalPages&gt;0&lt;/TotalPages&gt;
    &lt;PageNumber&gt;1&lt;/PageNumber&gt;
&lt;/result&gt; 
</code></pre></div>
<h2 dir="auto">Reproduction Steps</h2>
<p dir="auto">This is the URL I am hitting:</p>
<p dir="auto"><code class="notranslate">http://productsearch.linksynergy.com/productsearch?token=&lt;token&gt;&amp;mid=36145&amp;cat=Women%27s+Warehouse+Sale+-+Up+To+70%+Off&amp;keyword=dress&amp;pagenumber=1</code></p>
<p dir="auto">Unfortunately, I cannot share the token. The decoded value for the <code class="notranslate">cat</code> parameter is "Women's Warehouse Sale - Up To 70% Off". The percent sign after 70 is the one that is causing the issue.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import requests

with requests.Session() as http_session:
     response = http_session.get('http://productsearch.linksynergy.com/productsearch', params='token=&lt;token&gt;&amp;mid=36145&amp;cat=Women%27s+Warehouse+Sale+-+Up+To+70%+Off&amp;keyword=dress&amp;pagenumber=1')"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">requests</span>

<span class="pl-k">with</span> <span class="pl-s1">requests</span>.<span class="pl-v">Session</span>() <span class="pl-k">as</span> <span class="pl-s1">http_session</span>:
     <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">http_session</span>.<span class="pl-en">get</span>(<span class="pl-s">'http://productsearch.linksynergy.com/productsearch'</span>, <span class="pl-s1">params</span><span class="pl-c1">=</span><span class="pl-s">'token=&lt;token&gt;&amp;mid=36145&amp;cat=Women%27s+Warehouse+Sale+-+Up+To+70%+Off&amp;keyword=dress&amp;pagenumber=1'</span>)</pre></div>
<h2 dir="auto">System Information</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ python -m requests.help"><pre class="notranslate"><code class="notranslate">$ python -m requests.help
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;chardet&quot;: {
    &quot;version&quot;: &quot;4.0.0&quot;
  },
  &quot;cryptography&quot;: {
    &quot;version&quot;: &quot;&quot;
  },
  &quot;idna&quot;: {
    &quot;version&quot;: &quot;2.10&quot;
  },
  &quot;implementation&quot;: {
    &quot;name&quot;: &quot;CPython&quot;,
    &quot;version&quot;: &quot;3.7.2&quot;
  },
  &quot;platform&quot;: {
    &quot;release&quot;: &quot;5.8.0-40-generic&quot;,
    &quot;system&quot;: &quot;Linux&quot;
  },
  &quot;pyOpenSSL&quot;: {
    &quot;openssl_version&quot;: &quot;&quot;,
    &quot;version&quot;: null
  },
  &quot;requests&quot;: {
    &quot;version&quot;: &quot;2.25.1&quot;
  },
  &quot;system_ssl&quot;: {
    &quot;version&quot;: &quot;1010106f&quot;
  },
  &quot;urllib3&quot;: {
    &quot;version&quot;: &quot;1.26.3&quot;
  },
  &quot;using_pyopenssl&quot;: false
}"><pre class="notranslate"><code class="notranslate">{
  "chardet": {
    "version": "4.0.0"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.2"
  },
  "platform": {
    "release": "5.8.0-40-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.25.1"
  },
  "system_ssl": {
    "version": "1010106f"
  },
  "urllib3": {
    "version": "1.26.3"
  },
  "using_pyopenssl": false
}
</code></pre></div>
<p dir="auto">This command is only available on Requests v2.16.4 and greater. Otherwise,<br>
please provide some basic information about your system (Python version,<br>
operating system, &amp;c).</p>
<p dir="auto">I'm using requests to download asynchronously and parse the content of a bunch of URLs from different hosts. When the number of URLs is small (up to 1000 URLs) all works fine. When I run the same script with 10000 I get the following error <strong>only when there are about 10 URLs left to download</strong>:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;many_async_req.py&quot;, line 35, in &lt;module&gt;
  File &quot;/home/alex/code/.virtualenvs/testreq/local/lib/python2.7/site-packages/requests-0.11.2-py2.7.egg/requests/async.py&quot;, line 83, in map
  File &quot;/home/alex/code/.virtualenvs/testreq/local/lib/python2.7/site-packages/gevent/greenlet.py&quot;, line 487, in joinall
ImportError: No module named queue"><pre class="notranslate"><span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>):
  <span class="pl-v">File</span> <span class="pl-s">"many_async_req.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">35</span>, <span class="pl-s1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>
  <span class="pl-v">File</span> <span class="pl-s">"/home/alex/code/.virtualenvs/testreq/local/lib/python2.7/site-packages/requests-0.11.2-py2.7.egg/requests/async.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">83</span>, <span class="pl-s1">in</span> <span class="pl-s1">map</span>
  <span class="pl-v">File</span> <span class="pl-s">"/home/alex/code/.virtualenvs/testreq/local/lib/python2.7/site-packages/gevent/greenlet.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">487</span>, <span class="pl-s1">in</span> <span class="pl-s1">joinall</span>
<span class="pl-v">ImportError</span>: <span class="pl-v">No</span> <span class="pl-s1">module</span> <span class="pl-s1">named</span> <span class="pl-s1">queue</span></pre></div>
<p dir="auto">Sometimes the script just hangs with about 10 URLs left. I observed this behavior with requests installed from pipy and with the code downloaded from the development branch.</p>
<p dir="auto">This happened to me using the version of gevent from pypi <strong>and</strong> using the <a href="http://code.google.com/p/gevent/downloads/list" rel="nofollow">latest version</a> downloaded from gevent's Google Code repository.<br>
I'm using a virtualenv with only the relevant modules installed:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="(testreq)alex@haiku:~/code$ pip freeze
certifi==0.0.8
chardet==1.0.1
distribute==0.6.19
gevent==0.13.7
greenlet==0.3.4
requests==0.11.2
wsgiref==0.1.2"><pre class="notranslate">(testreq)alex@haiku:<span class="pl-k">~</span>/code$ pip freeze
certifi==0.0.8
chardet==1.0.1
distribute==0.6.19
gevent==0.13.7
greenlet==0.3.4
requests==0.11.2
wsgiref==0.1.2</pre></div>
<p dir="auto">To reproduce the error I'm using the following script (that is a stripped-down version of the original) that basically just print the status code of each URL.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import codecs
import random
from requests import async
from requests import defaults
import sys

defaults.defaults['base_headers']['User-Agent'] = &quot;Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0&quot;
defaults.defaults['max_retries'] = 5
defaults.defaults['verbose'] = sys.stderr


def check(response):
    global counter
    print &quot;\t %s HTTP status code: %s&quot; % (counter, response.status_code)
    counter += 1


def load_urls(fname):
    &quot;&quot;&quot; Read URLs from file and remove duplicates &quot;&quot;&quot;
    with codecs.open(fname, 'r', 'utf-8') as fh:
        urls = list(set([url.strip() for url in fh.readlines()]))
        random.shuffle(urls)
        return urls

if __name__ == '__main__':

    urls = load_urls('urls.txt')
    limit = int(sys.argv[1])
    urls = urls[:limit]
    rs = []
    counter = 0
    todo = len(urls)
    for url in urls:
            rs.append(async.get(url, hooks=dict(response=check), timeout=5.0))
    async.map(rs, size=10)
    print &quot;To check: %s&quot; % todo
    print &quot;To checked: %s&quot; % counter"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">codecs</span>
<span class="pl-k">import</span> <span class="pl-s1">random</span>
<span class="pl-k">from</span> <span class="pl-s1">requests</span> <span class="pl-k">import</span> <span class="pl-s1">async</span>
<span class="pl-k">from</span> <span class="pl-s1">requests</span> <span class="pl-k">import</span> <span class="pl-s1">defaults</span>
<span class="pl-k">import</span> <span class="pl-s1">sys</span>

<span class="pl-s1">defaults</span>.<span class="pl-s1">defaults</span>[<span class="pl-s">'base_headers'</span>][<span class="pl-s">'User-Agent'</span>] <span class="pl-c1">=</span> <span class="pl-s">"Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0"</span>
<span class="pl-s1">defaults</span>.<span class="pl-s1">defaults</span>[<span class="pl-s">'max_retries'</span>] <span class="pl-c1">=</span> <span class="pl-c1">5</span>
<span class="pl-s1">defaults</span>.<span class="pl-s1">defaults</span>[<span class="pl-s">'verbose'</span>] <span class="pl-c1">=</span> <span class="pl-s1">sys</span>.<span class="pl-s1">stderr</span>


<span class="pl-k">def</span> <span class="pl-en">check</span>(<span class="pl-s1">response</span>):
    <span class="pl-k">global</span> <span class="pl-s1">counter</span>
    <span class="pl-k">print</span> <span class="pl-s">"<span class="pl-cce">\t</span> %s HTTP status code: %s"</span> <span class="pl-c1">%</span> (<span class="pl-s1">counter</span>, <span class="pl-s1">response</span>.<span class="pl-s1">status_code</span>)
    <span class="pl-s1">counter</span> <span class="pl-c1">+=</span> <span class="pl-c1">1</span>


<span class="pl-k">def</span> <span class="pl-en">load_urls</span>(<span class="pl-s1">fname</span>):
    <span class="pl-s">""" Read URLs from file and remove duplicates """</span>
    <span class="pl-k">with</span> <span class="pl-s1">codecs</span>.<span class="pl-en">open</span>(<span class="pl-s1">fname</span>, <span class="pl-s">'r'</span>, <span class="pl-s">'utf-8'</span>) <span class="pl-k">as</span> <span class="pl-s1">fh</span>:
        <span class="pl-s1">urls</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-en">set</span>([<span class="pl-s1">url</span>.<span class="pl-en">strip</span>() <span class="pl-k">for</span> <span class="pl-s1">url</span> <span class="pl-c1">in</span> <span class="pl-s1">fh</span>.<span class="pl-en">readlines</span>()]))
        <span class="pl-s1">random</span>.<span class="pl-en">shuffle</span>(<span class="pl-s1">urls</span>)
        <span class="pl-k">return</span> <span class="pl-s1">urls</span>

<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">'__main__'</span>:

    <span class="pl-s1">urls</span> <span class="pl-c1">=</span> <span class="pl-en">load_urls</span>(<span class="pl-s">'urls.txt'</span>)
    <span class="pl-s1">limit</span> <span class="pl-c1">=</span> <span class="pl-en">int</span>(<span class="pl-s1">sys</span>.<span class="pl-s1">argv</span>[<span class="pl-c1">1</span>])
    <span class="pl-s1">urls</span> <span class="pl-c1">=</span> <span class="pl-s1">urls</span>[:<span class="pl-s1">limit</span>]
    <span class="pl-s1">rs</span> <span class="pl-c1">=</span> []
    <span class="pl-s1">counter</span> <span class="pl-c1">=</span> <span class="pl-c1">0</span>
    <span class="pl-s1">todo</span> <span class="pl-c1">=</span> <span class="pl-en">len</span>(<span class="pl-s1">urls</span>)
    <span class="pl-k">for</span> <span class="pl-s1">url</span> <span class="pl-c1">in</span> <span class="pl-s1">urls</span>:
            <span class="pl-s1">rs</span>.<span class="pl-en">append</span>(<span class="pl-s1">async</span>.<span class="pl-en">get</span>(<span class="pl-s1">url</span>, <span class="pl-s1">hooks</span><span class="pl-c1">=</span><span class="pl-en">dict</span>(<span class="pl-s1">response</span><span class="pl-c1">=</span><span class="pl-s1">check</span>), <span class="pl-s1">timeout</span><span class="pl-c1">=</span><span class="pl-c1">5.0</span>))
    <span class="pl-s1">async</span>.<span class="pl-en">map</span>(<span class="pl-s1">rs</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)
    <span class="pl-k">print</span> <span class="pl-s">"To check: %s"</span> <span class="pl-c1">%</span> <span class="pl-s1">todo</span>
    <span class="pl-k">print</span> <span class="pl-s">"To checked: %s"</span> <span class="pl-c1">%</span> <span class="pl-s1">counter</span></pre></div>