<p dir="auto">Would it be possible to have a dependency against requests (Apache 2 licensed) in Scrapy?</p>
<p dir="auto">I noticed that it has superior proxy env handling, where urllib fails to parse IPs or CIDR in a <code class="notranslate">$no_proxy</code> variable. This has led to failing testcases for me and I'd like to fix it.</p>
<p dir="auto">Also, requests provides a nice <a href="https://github.com/kennethreitz/requests/blob/383b20b02324e5c5d1970dd3ab0ba29bd991d2b2/requests/compat.py#L35-L62"><code class="notranslate">compat</code> module</a>, wrapping urllib/2 for PY2 and PY3 differences.<br>
If not a dependency, could scrapy do this stuff similarly?<br>
Pulling in urllib/urllib2/urlparse from a local wrapper seems cleaner than throwing <code class="notranslate">six.moves.</code> and <code class="notranslate">try...except</code> statements everywhere we use them. And we have plenty of those imports.</p>
<p dir="auto">scrapy version: 1.1.2<br>
python version: 2.7.12<br>
platform: Mac OS X 10.11.6</p>
<p dir="auto"><strong>The issue:</strong></p>
<p dir="auto">For the url given in the following minimum working example, the HTML text in the response from scrapy is different from the one obtained with <code class="notranslate">requests.get</code>.   The latter seems to be the correct one.  It seems scrapy somehow duplicates part of the response html.  This does not happen for all sites.</p>
<p dir="auto">See the attached file for the two different html files.  Or you may run the following code to generate them.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import scrapy
from scrapy.crawler import CrawlerRunner
from twisted.internet import reactor
import requests

url = 'http://training.sac.net.cn/cms/flkcalone.htm?myId=4028d0ee57ec28180157f55059b87209'

headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, sdch',
    'Accept-Language': 'en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4,zh-TW;q=0.2',
    'Cache-Control': 'max-age=0',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36',
    }


class Test(scrapy.Spider):
    name = 'test'

    def start_requests(self):
        request = scrapy.Request(url=url, callback=self.parse, headers=headers)
        yield request

    def parse(self, response):
        with open('response_with_scrapy.html', 'w') as f:
            f.write(response.text.encode('utf-8'))


if __name__ == '__main__':
    with open('response_with_requests.html', 'w') as f:
        f.write(requests.get(url, headers=headers).text.encode('utf-8'))

    runner = CrawlerRunner()
    runner.crawl(Test)
    d = runner.join()
    d.addBoth(lambda _: reactor.stop())
    reactor.run()"><pre class="notranslate"><code class="notranslate">import scrapy
from scrapy.crawler import CrawlerRunner
from twisted.internet import reactor
import requests

url = 'http://training.sac.net.cn/cms/flkcalone.htm?myId=4028d0ee57ec28180157f55059b87209'

headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, sdch',
    'Accept-Language': 'en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4,zh-TW;q=0.2',
    'Cache-Control': 'max-age=0',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36',
    }


class Test(scrapy.Spider):
    name = 'test'

    def start_requests(self):
        request = scrapy.Request(url=url, callback=self.parse, headers=headers)
        yield request

    def parse(self, response):
        with open('response_with_scrapy.html', 'w') as f:
            f.write(response.text.encode('utf-8'))


if __name__ == '__main__':
    with open('response_with_requests.html', 'w') as f:
        f.write(requests.get(url, headers=headers).text.encode('utf-8'))

    runner = CrawlerRunner()
    runner.crawl(Test)
    d = runner.join()
    d.addBoth(lambda _: reactor.stop())
    reactor.run()
</code></pre></div>
<p dir="auto"><a href="https://github.com/scrapy/scrapy/files/639455/two_responses.zip">two_responses.zip</a></p>