<p dir="auto">Proxy <code class="notranslate">CONNECT</code> support is required for https urls to work.</p>
<p dir="auto">When proxy support was added to Scrapy, urllib still didn't supported CONNECT, but that has been fixed now:<br>
<a href="http://bugs.python.org/issue1424152" rel="nofollow">http://bugs.python.org/issue1424152</a></p>
<p dir="auto">Now we need to add support to Scrapy (which doesn't use urllib at all).</p>
<p dir="auto">This link contains useful tips on how to add CONNECT support with Twisted:<br>
<a href="http://twistedmatrix.com/pipermail/twisted-web/2008-August/003878.html" rel="nofollow">http://twistedmatrix.com/pipermail/twisted-web/2008-August/003878.html</a></p>
<p dir="auto">See originally reported ticket on Trac for more comments: <a href="http://dev.scrapy.org/ticket/159" rel="nofollow">http://dev.scrapy.org/ticket/159</a></p>
<p dir="auto">I have a spider that's throwing the following error when trying to crawl this URL.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; fetch('https://vconnections.org/resources')
2015-08-12 10:07:28 [scrapy] INFO: Spider opened
2015-08-12 10:07:28 [scrapy] DEBUG: Retrying &lt;GET https://vconnections.org/resources&gt; (failed 1 times): [&lt;twisted.python.failure.Failure &lt;class 'OpenSSL.SSL.Error'&gt;&gt;]
2015-08-12 10:07:33 [scrapy] DEBUG: Gave up retrying &lt;GET https://vconnections.org/resources&gt; (failed 2 times): [&lt;twisted.python.failure.Failure &lt;class 'OpenSSL.SSL.Error'&gt;&gt;]
Traceback (most recent call last):
  File &quot;&lt;console&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;/Users/gmeans/.virtualenvs/backlink/lib/python2.7/site-packages/scrapy/shell.py&quot;, line 87, in fetch
    reactor, self._schedule, request, spider)
  File &quot;/Users/gmeans/.virtualenvs/backlink/lib/python2.7/site-packages/twisted/internet/threads.py&quot;, line 122, in blockingCallFromThread
    result.raiseException()
  File &quot;&lt;string&gt;&quot;, line 2, in raiseException
ResponseNeverReceived: [&lt;twisted.python.failure.Failure &lt;class 'OpenSSL.SSL.Error'&gt;&gt;]"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">fetch</span>(<span class="pl-s">'https://vconnections.org/resources'</span>)
<span class="pl-c1">2015</span><span class="pl-c1">-</span><span class="pl-c1">08</span><span class="pl-c1">-</span><span class="pl-c1">12</span> <span class="pl-c1">10</span>:<span class="pl-c1">07</span>:<span class="pl-c1">28</span> [<span class="pl-s1">scrapy</span>] <span class="pl-v">INFO</span>: <span class="pl-v">Spider</span> <span class="pl-s1">opened</span>
<span class="pl-c1">2015</span><span class="pl-c1">-</span><span class="pl-c1">08</span><span class="pl-c1">-</span><span class="pl-c1">12</span> <span class="pl-c1">10</span>:<span class="pl-c1">07</span>:<span class="pl-c1">28</span> [<span class="pl-s1">scrapy</span>] <span class="pl-v">DEBUG</span>: <span class="pl-v">Retrying</span> <span class="pl-c1">&lt;</span><span class="pl-v">GET</span> <span class="pl-s1">https</span>:<span class="pl-c1">//</span><span class="pl-s1">vconnections</span>.<span class="pl-s1">org</span><span class="pl-c1">/</span><span class="pl-s1">resources</span><span class="pl-c1">&gt;</span> (<span class="pl-s1">failed</span> <span class="pl-c1">1</span> <span class="pl-s1">times</span>): [<span class="pl-c1">&lt;</span><span class="pl-s1">twisted</span>.<span class="pl-s1">python</span>.<span class="pl-s1">failure</span>.<span class="pl-v">Failure</span> <span class="pl-c1">&lt;</span><span class="pl-s1">class</span> <span class="pl-s">'OpenSSL.SSL.Error'</span><span class="pl-c1">&gt;&gt;</span>]
<span class="pl-c1">2015</span><span class="pl-c1">-</span><span class="pl-c1">08</span><span class="pl-c1">-</span><span class="pl-c1">12</span> <span class="pl-c1">10</span>:<span class="pl-c1">07</span>:<span class="pl-c1">33</span> [<span class="pl-s1">scrapy</span>] <span class="pl-v">DEBUG</span>: <span class="pl-v">Gave</span> <span class="pl-s1">up</span> <span class="pl-s1">retrying</span> <span class="pl-c1">&lt;</span><span class="pl-v">GET</span> <span class="pl-s1">https</span>:<span class="pl-c1">//</span><span class="pl-s1">vconnections</span>.<span class="pl-s1">org</span><span class="pl-c1">/</span><span class="pl-s1">resources</span><span class="pl-c1">&gt;</span> (<span class="pl-s1">failed</span> <span class="pl-c1">2</span> <span class="pl-s1">times</span>): [<span class="pl-c1">&lt;</span><span class="pl-s1">twisted</span>.<span class="pl-s1">python</span>.<span class="pl-s1">failure</span>.<span class="pl-v">Failure</span> <span class="pl-c1">&lt;</span><span class="pl-s1">class</span> <span class="pl-s">'OpenSSL.SSL.Error'</span><span class="pl-c1">&gt;&gt;</span>]
<span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>):
  <span class="pl-v">File</span> <span class="pl-s">"&lt;console&gt;"</span>, <span class="pl-s1">line</span> <span class="pl-c1">1</span>, <span class="pl-s1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>
  <span class="pl-v">File</span> <span class="pl-s">"/Users/gmeans/.virtualenvs/backlink/lib/python2.7/site-packages/scrapy/shell.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">87</span>, <span class="pl-s1">in</span> <span class="pl-s1">fetch</span>
    <span class="pl-s1">reactor</span>, <span class="pl-s1">self</span>.<span class="pl-s1">_schedule</span>, <span class="pl-s1">request</span>, <span class="pl-s1">spider</span>)
  <span class="pl-v">File</span> <span class="pl-s">"/Users/gmeans/.virtualenvs/backlink/lib/python2.7/site-packages/twisted/internet/threads.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">122</span>, <span class="pl-s1">in</span> <span class="pl-s1">blockingCallFromThread</span>
    <span class="pl-s1">result</span>.<span class="pl-en">raiseException</span>()
  <span class="pl-v">File</span> <span class="pl-s">"&lt;string&gt;"</span>, <span class="pl-s1">line</span> <span class="pl-c1">2</span>, <span class="pl-s1">in</span> <span class="pl-s1">raiseException</span>
<span class="pl-v">ResponseNeverReceived</span>: [<span class="pl-c1">&lt;</span><span class="pl-s1">twisted</span>.<span class="pl-s1">python</span>.<span class="pl-s1">failure</span>.<span class="pl-v">Failure</span> <span class="pl-c1">&lt;</span><span class="pl-s1">class</span> <span class="pl-s">'OpenSSL.SSL.Error'</span><span class="pl-c1">&gt;&gt;</span>]</pre></div>
<p dir="auto">Other SSL urls work fine, and I tried implementing the solution from this previous issue:</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="51783745" data-permission-text="Title is private" data-url="https://github.com/scrapy/scrapy/issues/981" data-hovercard-type="issue" data-hovercard-url="/scrapy/scrapy/issues/981/hovercard" href="https://github.com/scrapy/scrapy/issues/981">#981</a></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="class CustomContextFactory(ScrapyClientContextFactory):
    def getContext(self, hostname=None, port=None):
        ctx = ClientContextFactory.getContext(self)
        # Enable all workarounds to SSL bugs as documented by
        # http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html
        ctx.set_options(SSL.OP_ALL)
        if hostname:
            ClientTLSOptions(hostname, ctx)
        return ctx"><pre class="notranslate"><span class="pl-k">class</span> <span class="pl-v">CustomContextFactory</span>(<span class="pl-v">ScrapyClientContextFactory</span>):
    <span class="pl-k">def</span> <span class="pl-en">getContext</span>(<span class="pl-s1">self</span>, <span class="pl-s1">hostname</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">port</span><span class="pl-c1">=</span><span class="pl-c1">None</span>):
        <span class="pl-s1">ctx</span> <span class="pl-c1">=</span> <span class="pl-v">ClientContextFactory</span>.<span class="pl-en">getContext</span>(<span class="pl-s1">self</span>)
        <span class="pl-c"># Enable all workarounds to SSL bugs as documented by</span>
        <span class="pl-c"># http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html</span>
        <span class="pl-s1">ctx</span>.<span class="pl-en">set_options</span>(<span class="pl-v">SSL</span>.<span class="pl-v">OP_ALL</span>)
        <span class="pl-k">if</span> <span class="pl-s1">hostname</span>:
            <span class="pl-v">ClientTLSOptions</span>(<span class="pl-s1">hostname</span>, <span class="pl-s1">ctx</span>)
        <span class="pl-k">return</span> <span class="pl-s1">ctx</span></pre></div>
<p dir="auto">Scrapy==1.0.3<br>
Twisted==15.3.0<br>
pyOpenSSL==0.15.1</p>
<p dir="auto">OpenSSL 1.0.1k 8 Jan 2015</p>
<p dir="auto">Any ideas on what else I could try? Thanks!</p>