<p dir="auto">Hello,</p>
<p dir="auto">I run scrapy v1.1.0</p>
<p dir="auto">I need some parts of the url to create some ids. Lately I have been noticing that response.request.url is returning a different url that the one that is used to scrape.</p>
<p dir="auto">In my code I use the item['LINK'] = response.request.url but that is putting in the wrong link on some scrapers like 2/10 approximately.</p>
<p dir="auto">Is there a better way to fetch the url? The one that is displayed in the debug scrapy output is correct.</p>
<p dir="auto">Thank you.</p>
<p dir="auto">This one outputted in the debug window is correct:<br>
2016-07-12 13:25:34 [scrapy] DEBUG: Scraped from &lt;200 XXX &gt;</p>
<p dir="auto">The one I fetch with response.request.url is different and wrong.</p>
<h3 dir="auto">Description</h3>
<p dir="auto">After running the scrapy shell with the defined url, I am getting the attribute error showing the following error: AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'</p>
<p dir="auto">scrapy shell "<a href="https://quotes.toscrape.com/tag/humor/" rel="nofollow">https://quotes.toscrape.com/tag/humor/</a>"</p>
<h3 dir="auto">Steps to Reproduce</h3>
<ol dir="auto">
<li>[First Step]</li>
<li>[Second Step]</li>
<li>[and so on...]</li>
</ol>
<p dir="auto"><strong>Expected behavior:</strong> [What you expect to happen]</p>
<p dir="auto"><strong>Actual behavior:</strong><br>
[2022-09-27 01:38:38 [scrapy.utils.log] INFO: Versions: lxml 4.9.1.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 2.0.1, Twisted 22.8.0, Python 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)], pyOpenSSL 22.1.0 (OpenSSL 3.0.5 5 Jul 2022), cryptography 38.0.1, Platform Windows-10-10.0.19041-SP0<br>
2022-09-27 01:38:38 [scrapy.crawler] INFO: Overridden settings:<br>
{'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter',<br>
'LOGSTATS_INTERVAL': 0}<br>
2022-09-27 01:38:38 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor<br>
2022-09-27 01:38:38 [scrapy.extensions.telnet] INFO: Telnet Password: d760ab2d8573ec57<br>
2022-09-27 01:38:38 [scrapy.middleware] INFO: Enabled extensions:<br>
['scrapy.extensions.corestats.CoreStats',<br>
'scrapy.extensions.telnet.TelnetConsole']<br>
2022-09-27 01:38:39 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.http.HTTPDownloadHandler" for scheme "http"<br>
Traceback (most recent call last):<br>
File "d:\python\scraping-project\scrapy\venv\lib\site-packages\scrapy\core\downloader\handlers_<em>init</em>_.py", line 49, in <em>load_handler<br>
dhcls = load_object(path)<br>
File "d:\python\scraping-project\scrapy\venv\lib\site-packages\scrapy\utils\misc.py", line 61, in load_object<br>
mod = import_module(module)<br>
File "C:\Users\DELL\AppData\Local\Programs\Python\Python39\lib\importlib_<em>init</em></em>.py", line 127, in import_module<br>
return _bootstrap._gcd_import(name[level:], package, level)<br>
File "", line 1030, in _gcd_import<br>
File "", line 1007, in _find_and_load<br>
File "", line 986, in _find_and_load_unlocked<br>
File "", line 680, in _load_unlocked<br>
File "", line 790, in exec_module<br>
File "", line 228, in _call_with_frames_removed<br>
File "d:\python\scraping-project\scrapy\venv\lib\site-packages\scrapy\core\downloader\handlers\http.py", line 2, in <br>
from scrapy.core.downloader.handlers.http11 import (<br>
File "d:\python\scraping-project\scrapy\venv\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 23, in <br>
from scrapy.core.downloader.contextfactory import load_context_factory_from_settings<br>
File "d:\python\scraping-project\scrapy\venv\lib\site-packages\scrapy\core\downloader\contextfactory.py", line 11, in <br>
from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions<br>
File "d:\python\scraping-project\scrapy\venv\lib\site-packages\scrapy\core\downloader\tls.py", line 23, in <br>
METHOD_SSLv3: SSL.SSLv3_METHOD,                     # SSL 3 (NOT recommended)<br>
AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD']</p>
<p dir="auto"><strong>Reproduces how often:</strong> [100%]</p>
<h3 dir="auto">Versions</h3>
<p dir="auto">Scrapy       : 2.6.2<br>
lxml         : 4.9.1.0<br>
libxml2      : 2.9.12<br>
cssselect    : 1.1.0<br>
parsel       : 1.6.0<br>
w3lib        : 2.0.1<br>
Twisted      : 22.8.0<br>
Python       : 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)]<br>
pyOpenSSL    : 22.1.0 (OpenSSL 3.0.5 5 Jul 2022)<br>
cryptography : 37.0.4<br>
Platform     : Windows-10-10.0.19041-SP0</p>
<h3 dir="auto">Additional context</h3>
<p dir="auto">Any additional information, configuration, data or output from commands that might be necessary to reproduce or understand the issue. Please try not to include screenshots of code or the command line, paste the contents as text instead. You can use <a href="https://help.github.com/en/articles/creating-and-highlighting-code-blocks">GitHub Flavored Markdown</a> to make the text look better.</p>