<h5 dir="auto">ISSUE TYPE</h5>
<ul dir="auto">
<li>Bug Report</li>
</ul>
<h5 dir="auto">COMPONENT NAME</h5>
<p dir="auto"><code class="notranslate">get_url</code> module</p>
<h5 dir="auto">ANSIBLE VERSION</h5>
<p dir="auto">Ansible 2.4.1.0 (sorry, don't have the exact ansible --version output at hand right now)</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="        &quot;ansible_python&quot;: {
            &quot;executable&quot;: &quot;/usr/bin/python&quot;,
            &quot;has_sslcontext&quot;: true,
            &quot;type&quot;: &quot;CPython&quot;,
            &quot;version&quot;: {
                &quot;major&quot;: 2,
                &quot;micro&quot;: 12,
                &quot;minor&quot;: 7,
                &quot;releaselevel&quot;: &quot;final&quot;,
                &quot;serial&quot;: 0
            },
            &quot;version_info&quot;: [
                2,
                7,
                12,
                &quot;final&quot;,
                0
            ]
        }"><pre class="notranslate">        <span class="pl-ent">"ansible_python"</span>: {
            <span class="pl-ent">"executable"</span>: <span class="pl-s"><span class="pl-pds">"</span>/usr/bin/python<span class="pl-pds">"</span></span>,
            <span class="pl-ent">"has_sslcontext"</span>: <span class="pl-c1">true</span>,
            <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>CPython<span class="pl-pds">"</span></span>,
            <span class="pl-ent">"version"</span>: {
                <span class="pl-ent">"major"</span>: <span class="pl-c1">2</span>,
                <span class="pl-ent">"micro"</span>: <span class="pl-c1">12</span>,
                <span class="pl-ent">"minor"</span>: <span class="pl-c1">7</span>,
                <span class="pl-ent">"releaselevel"</span>: <span class="pl-s"><span class="pl-pds">"</span>final<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"serial"</span>: <span class="pl-c1">0</span>
            },
            <span class="pl-ent">"version_info"</span>: [
                <span class="pl-c1">2</span>,
                <span class="pl-c1">7</span>,
                <span class="pl-c1">12</span>,
                <span class="pl-s"><span class="pl-pds">"</span>final<span class="pl-pds">"</span></span>,
                <span class="pl-c1">0</span>
            ]
        }</pre></div>
<h5 dir="auto">CONFIGURATION</h5>
<p dir="auto">nothing changed</p>
<h5 dir="auto">OS / ENVIRONMENT</h5>
<p dir="auto">Ubuntu 16.04 in a Vagrant Box running on VirtualBox on Windows. The script runs during Vagrant's Ansible provisioning.</p>
<h5 dir="auto">SUMMARY</h5>
<p dir="auto">Ansible's get_url module fails to retrieve data when these conditions are met:</p>
<ul dir="auto">
<li>A proxy is defined (through https_proxy environment variable in my case)</li>
<li>The target url is secure i.e. uses https://</li>
<li>Certificate validation is active (which it is by default)</li>
</ul>
<p dir="auto">It then says "Connection to proxy failed". In my case there's a cascade of proxies with a local cntlm talking to the corporate squid proxy.</p>
<p dir="auto">The problem does not occur when you use <code class="notranslate">validate_certs: no</code>. In Ansible the issues seems to arise from </p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/ansible/ansible/blob/e946c4b3b73be28871ab67965584e060d7da1c4b/lib/ansible/module_utils/urls.py#L602">ansible/lib/ansible/module_utils/urls.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 602
      in
      <a data-pjax="true" class="commit-tease-sha" href="/ansible/ansible/commit/e946c4b3b73be28871ab67965584e060d7da1c4b">e946c4b</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L602" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="602"></td>
          <td id="LC602" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-v">CONNECT_COMMAND</span> <span class="pl-c1">=</span> <span class="pl-s">"CONNECT %s:%s HTTP/1.0<span class="pl-cce">\r</span><span class="pl-cce">\n</span>Connection: close<span class="pl-cce">\r</span><span class="pl-cce">\n</span>"</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
 That is part of the <code class="notranslate">SSLValidationHandler</code> in that file. And that is only used when using <code class="notranslate">validate_certs: yes</code>. Please see below for details and packet captures.<p></p>
<p dir="auto">The problem is pretty much the same as <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="257425810" data-permission-text="Title is private" data-url="https://github.com/request/request/issues/2766" data-hovercard-type="issue" data-hovercard-url="/request/request/issues/2766/hovercard" href="https://github.com/request/request/issues/2766">request/request#2766</a> in the <code class="notranslate">requests</code> library. Similar problems have been reported in Ansible before (see <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="73961578" data-permission-text="Title is private" data-url="https://github.com/ansible/ansible/issues/10941" data-hovercard-type="issue" data-hovercard-url="/ansible/ansible/issues/10941/hovercard" href="https://github.com/ansible/ansible/issues/10941">#10941</a>) but that discussion is a bit misleading and eventually led nowhere.</p>
<h5 dir="auto">STEPS TO REPRODUCE</h5>
<p dir="auto">You'd need a web proxy like squid. Make sure to <code class="notranslate">export https_proxy=http://whatever-adress-your-proxy-may-have.net:3128</code> to point to that proxy.</p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="    - name: download nodejs script to add repository
      get_url:
        url: https://deb.nodesource.com/setup_7.x
        dest: /tmp/nodejs_setup_7.x
        mode: 0755"><pre class="notranslate">    - <span class="pl-ent">name</span>: <span class="pl-s">download nodejs script to add repository</span>
      <span class="pl-ent">get_url</span>:
        <span class="pl-ent">url</span>: <span class="pl-s">https://deb.nodesource.com/setup_7.x</span>
        <span class="pl-ent">dest</span>: <span class="pl-s">/tmp/nodejs_setup_7.x</span>
        <span class="pl-ent">mode</span>: <span class="pl-c1">0755</span></pre></div>

<h5 dir="auto">EXPECTED RESULTS</h5>
<p dir="auto">get_url should connect through the proxy, verify the certificate, fetch the URL, and save the file.</p>
<h5 dir="auto">ACTUAL RESULTS</h5>
<p dir="auto">The file is not retrieved. An error is written and the below traceback is printed:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  File &quot;/tmp/ansible_akRnA9/ansible_modlib.zip/ansible/module_utils/urls.py&quot;, line 1044, in fetch_url
    client_key=client_key, cookies=cookies)
  File &quot;/tmp/ansible_akRnA9/ansible_modlib.zip/ansible/module_utils/urls.py&quot;, line 951, in open_url
    r = urllib_request.urlopen(*urlopen_args)
  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 154, in urlopen
    return opener.open(url, data, timeout)
  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 427, in open
    req = meth(req)
  File &quot;/tmp/ansible_akRnA9/ansible_modlib.zip/ansible/module_utils/urls.py&quot;, line 740, in http_request
    self.validate_proxy_response(connect_result)
  File &quot;/tmp/ansible_akRnA9/ansible_modlib.zip/ansible/module_utils/urls.py&quot;, line 682, in validate_proxy_response
    raise ProxyError('Connection to proxy failed')"><pre class="notranslate"><code class="notranslate">  File "/tmp/ansible_akRnA9/ansible_modlib.zip/ansible/module_utils/urls.py", line 1044, in fetch_url
    client_key=client_key, cookies=cookies)
  File "/tmp/ansible_akRnA9/ansible_modlib.zip/ansible/module_utils/urls.py", line 951, in open_url
    r = urllib_request.urlopen(*urlopen_args)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 427, in open
    req = meth(req)
  File "/tmp/ansible_akRnA9/ansible_modlib.zip/ansible/module_utils/urls.py", line 740, in http_request
    self.validate_proxy_response(connect_result)
  File "/tmp/ansible_akRnA9/ansible_modlib.zip/ansible/module_utils/urls.py", line 682, in validate_proxy_response
    raise ProxyError('Connection to proxy failed')
</code></pre></div>
<p dir="auto">I used tcpdump to compare get_url with</p>
<ul dir="auto">
<li><code class="notranslate">validate_certs: yes</code></li>
<li><code class="notranslate">validate_certs: no</code></li>
<li>and <code class="notranslate">wget</code></li>
</ul>
<p dir="auto">This is the result in packet capture snippets:<br>
<strong>validate yes</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="CONNECT deb.nodesource.com:443 HTTP/1.0
Connection: close


HTTP/1.1 407 Proxy Authentication Required
Server: squid/3.5.27
Mime-Version: 1.0
Date: Wed, 08 Nov 2017 07:25:21 GMT"><pre class="notranslate"><code class="notranslate">CONNECT deb.nodesource.com:443 HTTP/1.0
Connection: close


HTTP/1.1 407 Proxy Authentication Required
Server: squid/3.5.27
Mime-Version: 1.0
Date: Wed, 08 Nov 2017 07:25:21 GMT
</code></pre></div>
<p dir="auto"><strong>validate no</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="CONNECT deb.nodesource.com:443 HTTP/1.0


HTTP/1.1 200 Connection established
Connection: close"><pre class="notranslate"><code class="notranslate">CONNECT deb.nodesource.com:443 HTTP/1.0


HTTP/1.1 200 Connection established
Connection: close
</code></pre></div>
<p dir="auto"><strong>wget</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="CONNECT deb.nodesource.com:443 HTTP/1.1
User-Agent: Wget/1.17.1 (linux-gnu)
Host: deb.nodesource.com:443


HTTP/1.1 200 Connection established
Connection: close"><pre class="notranslate"><code class="notranslate">CONNECT deb.nodesource.com:443 HTTP/1.1
User-Agent: Wget/1.17.1 (linux-gnu)
Host: deb.nodesource.com:443


HTTP/1.1 200 Connection established
Connection: close
</code></pre></div>
<p dir="auto">Notice that when using certificate validation get_url sets the <code class="notranslate">Connection: close</code> header. This causes squid to kill the connection right away. There seems to be a violation of <a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html" rel="nofollow">rfc 2616</a>.</p>
<p dir="auto">I wouldn't know what good it does anyway to set the header the way it's done. So I suggest to just get rid of setting the header. After all, it's not set when validation is off.</p>
<p dir="auto">I want to assert that certain EC2 security groups are present.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="- name: Create EC2 SSH security group.
  ec2_group:
    name: SSH
    description: Allows SSH access.
    state: present
    region: &quot;{{ aws_region }}&quot;
    rules:
      - proto: tcp 
        from_port: 22
        to_port: 22
        cidr_ip: 0.0.0.0/0"><pre class="notranslate"><code class="notranslate">- name: Create EC2 SSH security group.
  ec2_group:
    name: SSH
    description: Allows SSH access.
    state: present
    region: "{{ aws_region }}"
    rules:
      - proto: tcp 
        from_port: 22
        to_port: 22
        cidr_ip: 0.0.0.0/0
</code></pre></div>
<p dir="auto">It works the first time:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ ansible-playbook -i hosts provision.yml 

PLAY [localhost] ************************************************************** 

TASK: [ec2 | Create EC2 SSH security group.] ********************************** 
changed: [localhost]

PLAY RECAP ******************************************************************** 
localhost                  : ok=1    changed=1    unreachable=0    failed=0"><pre class="notranslate"><code class="notranslate">$ ansible-playbook -i hosts provision.yml 

PLAY [localhost] ************************************************************** 

TASK: [ec2 | Create EC2 SSH security group.] ********************************** 
changed: [localhost]

PLAY RECAP ******************************************************************** 
localhost                  : ok=1    changed=1    unreachable=0    failed=0
</code></pre></div>
<p dir="auto">But when running it again, it fails:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ ansible-playbook -i hosts provision.yml 

PLAY [localhost] ************************************************************** 

TASK: [ec2 | Create EC2 SSH security group.] ********************************** 
failed: [localhost] =&gt; {&quot;failed&quot;: true, &quot;item&quot;: &quot;&quot;, &quot;parsed&quot;: false}
invalid output was: Traceback (most recent call last):
  File &quot;/home/danilo/.ansible/tmp/ansible-1389865891.33-218682145301746/ec2_group&quot;, line 1313, in &lt;module&gt;
    main()
  File &quot;/home/danilo/.ansible/tmp/ansible-1389865891.33-218682145301746/ec2_group&quot;, line 192, in main
    group = ec2.create_security_group(name, description, vpc_id=vpc_id)
  File &quot;/usr/lib/python2.7/site-packages/boto/ec2/connection.py&quot;, line 2948, in create_security_group
    SecurityGroup, verb='POST')
  File &quot;/usr/lib/python2.7/site-packages/boto/connection.py&quot;, line 1151, in get_object
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
&lt;Response&gt;&lt;Errors&gt;&lt;Error&gt;&lt;Code&gt;InvalidGroup.Duplicate&lt;/Code&gt;&lt;Message&gt;The security group 'SSH' already exists for VPC 'vpc-ab1e15c9'&lt;/Message&gt;&lt;/Error&gt;&lt;/Errors&gt;&lt;RequestID&gt;76a8ffae-6e58-4ab4-aacb-fff220476816&lt;/RequestID&gt;&lt;/Response&gt;


FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/home/danilo/provision.retry

localhost                  : ok=0    changed=0    unreachable=0    failed=1"><pre class="notranslate"><code class="notranslate">$ ansible-playbook -i hosts provision.yml 

PLAY [localhost] ************************************************************** 

TASK: [ec2 | Create EC2 SSH security group.] ********************************** 
failed: [localhost] =&gt; {"failed": true, "item": "", "parsed": false}
invalid output was: Traceback (most recent call last):
  File "/home/danilo/.ansible/tmp/ansible-1389865891.33-218682145301746/ec2_group", line 1313, in &lt;module&gt;
    main()
  File "/home/danilo/.ansible/tmp/ansible-1389865891.33-218682145301746/ec2_group", line 192, in main
    group = ec2.create_security_group(name, description, vpc_id=vpc_id)
  File "/usr/lib/python2.7/site-packages/boto/ec2/connection.py", line 2948, in create_security_group
    SecurityGroup, verb='POST')
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1151, in get_object
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;Response&gt;&lt;Errors&gt;&lt;Error&gt;&lt;Code&gt;InvalidGroup.Duplicate&lt;/Code&gt;&lt;Message&gt;The security group 'SSH' already exists for VPC 'vpc-ab1e15c9'&lt;/Message&gt;&lt;/Error&gt;&lt;/Errors&gt;&lt;RequestID&gt;76a8ffae-6e58-4ab4-aacb-fff220476816&lt;/RequestID&gt;&lt;/Response&gt;


FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/home/danilo/provision.retry

localhost                  : ok=0    changed=0    unreachable=0    failed=1
</code></pre></div>
<p dir="auto">Is there a way to make this module idempotent? When a rule with the same name already exists, the rules should be compared. If they're the same, it should set the status to "not changed". (Otherwise an error could be thrown.)</p>