<p dir="auto">I worked through an example from the <a href="http://doc.scrapy.org/en/latest/topics/shell.html" rel="nofollow">documentation</a> on the scrapy shell. One of the commands there did not work however and I got an <code class="notranslate">AlreadyCalledError</code> from a deferred in Twisted.</p>
<p dir="auto">More specifically, <a href="http://doc.scrapy.org/en/latest/topics/shell.html#example-of-shell-session" rel="nofollow">this example</a> failed during the call to <code class="notranslate">request.replace(method="POST")</code>. The method should return a new object according to its doc, but I think the reason Twisted throws the <code class="notranslate">AlreadyCalledError</code> is because it has run through the deferred's callbacks already in the new object. In other words, <code class="notranslate">request.replace()</code> is not returning an independent clone.</p>
<p dir="auto">Here's a <a href="https://gist.github.com/gkb/c55c21316ccb2355d792">gist</a> containing the exact error message.</p>
<p dir="auto">I try to get items from selector, method xpath consider my xpath as a string. I have seen in a book that if i want to iterate over my selector instead of the all page i should use a dot before my xpath, but it seems that doesn't work.  This cause i can't get the URL of the article to continue to add items in the items loader from the method parse_article_text. In parse_article_text, i think i don't want to use selector anymore i can use the response to get items from the all page, because i am in the full article page.</p>
<h2 dir="auto">Example</h2>
<p dir="auto">I want to get infos from this selector --&gt; article_xpath = "//div[<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/id/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/id">@id</a>='zox-feat-wrap' or <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/id/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/id">@id</a>='zox-home-cont-wrap']//*[contains(<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Class/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Class">@Class</a>,'zox-art-wrap zoxrel')]"<br>
Try to get the title from the selector, this why i use a dot in --&gt; titre_article_xpath = ".//h2[<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Class/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Class">@Class</a>='zox-s-title2' or <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Class/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Class">@Class</a>='zox-s-title1']/text()"</p>
<p dir="auto">But the true xpath for all the items in the method parse_article begin with //div[<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/id/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/id">@id</a>='zox-feat-wrap' or <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/id/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/id">@id</a>='zox-home-cont-wrap']//*[contains(<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Class/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Class">@Class</a>,'zox-art-wrap zoxrel')]</p>
<p dir="auto">Ex: the full xpath for the titre_article_xpath mentioned before is //div[<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/id/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/id">@id</a>='zox-feat-wrap' or <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/id/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/id">@id</a>='zox-home-cont-wrap']//*[contains(<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Class/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Class">@Class</a>,'zox-art-wrap zoxrel')]//h2[<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Class/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Class">@Class</a>='zox-s-title2' or <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Class/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Class">@Class</a>='zox-s-title1']/text()</p>
<p dir="auto">`import scrapy<br>
from ..items import HapiItem<br>
from scrapy.loader import ItemLoader<br>
from itemloaders.processors import MapCompose, Join, Compose<br>
from w3lib.html import replace_escape_chars</p>
<p dir="auto">class Juno7Spider(scrapy.Spider):<br>
name = 'juno7'<br>
allowed_domains = ['juno7.ht']</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="url = 'https://www.juno7.ht/{}'
list_category = ('sports/', 'politique/')


def start_requests(self):

    for category in Juno7Spider.list_category:

        url_category = Juno7Spider.url.format(category)
        yield scrapy.Request(url_category, self.parse)

def parse(self, response):
    
    suivant_url_xpath = &quot;//a[normalize-space()='Next ›']/@href&quot;


    article_xpath = &quot;//div[@id='zox-feat-wrap' or @id='zox-home-cont-wrap']//*[contains(@class,'zox-art-wrap zoxrel')]&quot;

    for article in response.xpath(article_xpath):
        yield self.parse_article(selector=article, response=response)

    next_page = response.xpath(suivant_url_xpath).get()

    if next_page is not None:
        yield response.follow(url=next_page, callback=self.parse)

def parse_article(self, selector=None, response=None):
    l = ItemLoader(item=HapiItem(), selector=selector)
    l.default_input_processor = MapCompose(lambda v: v.split(), replace_escape_chars)
    l.default_output_processor = Join()

    lien_article_xpath = &quot;.//div[@class='zox-art-title']/a/@href&quot;

    titre_article_xpath = &quot;.//h2[@class='zox-s-title2' or @class='zox-s-title1']/text()&quot;

    date_article_xpath = &quot;.//div[@class='zox-byline-wrap']/span[@class='zox-byline-date']/text()&quot;  # format 8 AVRIL 2021 dd mm yyyy or time AGO jusqu'au 7 jours ex 7 JOURS AGO
    image_couverture_article_url_xpath = &quot;.//img[contains(@class,'zox-reg-img')]/@src&quot;

    auteur_article_xpath = &quot;.//div[@class='zox-byline-wrap']/span[@class='zox-byline-name']/text()&quot;
    category_article_xpath = &quot;.//h3[@class='zox-s-cat']/span/text()&quot;

    l.add_xpath('title', titre_article_xpath)
    l.add_xpath('url_cover_image', image_couverture_article_url_xpath,
                MapCompose(lambda i: response.urljoin(i)))
    l.add_value('author', auteur_article_xpath)

    l.add_xpath('date', date_article_xpath)
    l.add_value('category', category_article_xpath)

    # return l.load_item()
    url = response.xpath(lien_article_xpath).get()
    return response.follow(url=url, callback=self.parse_article_text, cb_kwargs=dict(item=l))

    #yield response.follow(url=lien_article_xpath, callback=self.parse_article_text, cb_kwargs=dict(item = l))

def parse_article_text(self, response, item):


    texte_article = &quot;//div[contains(@class, 'zox-post-body left zoxrel zox100')]//div[@class='theiaPostSlider_preloadedSlide']/p[position() &lt; (count(//h5[text()='En savoir plus:']/preceding-sibling::*))]/text()&quot;   
    item.add_xpath('text', texte_article)
    item.add_value('url_article', response.url)
    return item.load_item()"><pre class="notranslate"><code class="notranslate">url = 'https://www.juno7.ht/{}'
list_category = ('sports/', 'politique/')


def start_requests(self):

    for category in Juno7Spider.list_category:

        url_category = Juno7Spider.url.format(category)
        yield scrapy.Request(url_category, self.parse)

def parse(self, response):
    
    suivant_url_xpath = "//a[normalize-space()='Next ›']/@href"


    article_xpath = "//div[@id='zox-feat-wrap' or @id='zox-home-cont-wrap']//*[contains(@class,'zox-art-wrap zoxrel')]"

    for article in response.xpath(article_xpath):
        yield self.parse_article(selector=article, response=response)

    next_page = response.xpath(suivant_url_xpath).get()

    if next_page is not None:
        yield response.follow(url=next_page, callback=self.parse)

def parse_article(self, selector=None, response=None):
    l = ItemLoader(item=HapiItem(), selector=selector)
    l.default_input_processor = MapCompose(lambda v: v.split(), replace_escape_chars)
    l.default_output_processor = Join()

    lien_article_xpath = ".//div[@class='zox-art-title']/a/@href"

    titre_article_xpath = ".//h2[@class='zox-s-title2' or @class='zox-s-title1']/text()"

    date_article_xpath = ".//div[@class='zox-byline-wrap']/span[@class='zox-byline-date']/text()"  # format 8 AVRIL 2021 dd mm yyyy or time AGO jusqu'au 7 jours ex 7 JOURS AGO
    image_couverture_article_url_xpath = ".//img[contains(@class,'zox-reg-img')]/@src"

    auteur_article_xpath = ".//div[@class='zox-byline-wrap']/span[@class='zox-byline-name']/text()"
    category_article_xpath = ".//h3[@class='zox-s-cat']/span/text()"

    l.add_xpath('title', titre_article_xpath)
    l.add_xpath('url_cover_image', image_couverture_article_url_xpath,
                MapCompose(lambda i: response.urljoin(i)))
    l.add_value('author', auteur_article_xpath)

    l.add_xpath('date', date_article_xpath)
    l.add_value('category', category_article_xpath)

    # return l.load_item()
    url = response.xpath(lien_article_xpath).get()
    return response.follow(url=url, callback=self.parse_article_text, cb_kwargs=dict(item=l))

    #yield response.follow(url=lien_article_xpath, callback=self.parse_article_text, cb_kwargs=dict(item = l))

def parse_article_text(self, response, item):


    texte_article = "//div[contains(@class, 'zox-post-body left zoxrel zox100')]//div[@class='theiaPostSlider_preloadedSlide']/p[position() &lt; (count(//h5[text()='En savoir plus:']/preceding-sibling::*))]/text()"   
    item.add_xpath('text', texte_article)
    item.add_value('url_article', response.url)
    return item.load_item()
</code></pre></div>
<p dir="auto">`</p>
<p dir="auto">**LOG **</p>
<p dir="auto">2021-05-05 03:10:42 [scrapy.core.engine] INFO: Spider opened<br>
2021-05-05 03:10:42 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)<br>
2021-05-05 03:10:42 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023<br>
2021-05-05 03:10:45 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/robots.txt%3E" rel="nofollow">https://www.juno7.ht/robots.txt&gt;</a> (referer: None)<br>
2021-05-05 03:10:46 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/sports/%3E" rel="nofollow">https://www.juno7.ht/sports/&gt;</a> (referer: None)<br>
2021-05-05 03:10:46 [scrapy.dupefilters] DEBUG: Filtered duplicate request: &lt;GET <a href="https://www.juno7.ht/jose-mourinho-ap-sou-ban-yon-kleb-italyen-selman-2022/%3E" rel="nofollow">https://www.juno7.ht/jose-mourinho-ap-sou-ban-yon-kleb-italyen-selman-2022/&gt;</a> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)<br>
2021-05-05 03:10:47 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/politique/%3E" rel="nofollow">https://www.juno7.ht/politique/&gt;</a> (referer: None)<br>
2021-05-05 03:10:47 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/sports/page/2/%3E" rel="nofollow">https://www.juno7.ht/sports/page/2/&gt;</a> (referer: <a href="https://www.juno7.ht/sports/" rel="nofollow">https://www.juno7.ht/sports/</a>)<br>
2021-05-05 03:10:48 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/politique/page/2/%3E" rel="nofollow">https://www.juno7.ht/politique/page/2/&gt;</a> (referer: <a href="https://www.juno7.ht/politique/" rel="nofollow">https://www.juno7.ht/politique/</a>)<br>
2021-05-05 03:10:49 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/sports/page/3/%3E" rel="nofollow">https://www.juno7.ht/sports/page/3/&gt;</a> (referer: <a href="https://www.juno7.ht/sports/page/2/" rel="nofollow">https://www.juno7.ht/sports/page/2/</a>)<br>
2021-05-05 03:10:50 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/politique/page/3/%3E" rel="nofollow">https://www.juno7.ht/politique/page/3/&gt;</a> (referer: <a href="https://www.juno7.ht/politique/page/2/" rel="nofollow">https://www.juno7.ht/politique/page/2/</a>)<br>
2021-05-05 03:10:50 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/sports/page/4/%3E" rel="nofollow">https://www.juno7.ht/sports/page/4/&gt;</a> (referer: <a href="https://www.juno7.ht/sports/page/3/" rel="nofollow">https://www.juno7.ht/sports/page/3/</a>)<br>
2021-05-05 03:10:51 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/politique/page/4/%3E" rel="nofollow">https://www.juno7.ht/politique/page/4/&gt;</a> (referer: <a href="https://www.juno7.ht/politique/page/3/" rel="nofollow">https://www.juno7.ht/politique/page/3/</a>)<br>
2021-05-05 03:10:52 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/sports/page/5/%3E" rel="nofollow">https://www.juno7.ht/sports/page/5/&gt;</a> (referer: <a href="https://www.juno7.ht/sports/page/4/" rel="nofollow">https://www.juno7.ht/sports/page/4/</a>)<br>
2021-05-05 03:10:53 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/politique/page/5/%3E" rel="nofollow">https://www.juno7.ht/politique/page/5/&gt;</a> (referer: <a href="https://www.juno7.ht/politique/page/4/" rel="nofollow">https://www.juno7.ht/politique/page/4/</a>)<br>
2021-05-05 03:10:53 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/sports/page/6/%3E" rel="nofollow">https://www.juno7.ht/sports/page/6/&gt;</a> (referer: <a href="https://www.juno7.ht/sports/page/5/" rel="nofollow">https://www.juno7.ht/sports/page/5/</a>)<br>
2021-05-05 03:10:54 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/politique/page/6/%3E" rel="nofollow">https://www.juno7.ht/politique/page/6/&gt;</a> (referer: <a href="https://www.juno7.ht/politique/page/5/" rel="nofollow">https://www.juno7.ht/politique/page/5/</a>)<br>
2021-05-05 03:10:55 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/sports/page/7/%3E" rel="nofollow">https://www.juno7.ht/sports/page/7/&gt;</a> (referer: <a href="https://www.juno7.ht/sports/page/6/" rel="nofollow">https://www.juno7.ht/sports/page/6/</a>)<br>
2021-05-05 03:10:55 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force<br>
2021-05-05 03:10:55 [scrapy.core.engine] INFO: Closing spider (shutdown)<br>
2021-05-05 03:10:55 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="https://www.juno7.ht/politique/page/7/%3E" rel="nofollow">https://www.juno7.ht/politique/page/7/&gt;</a> (referer: <a href="https://www.juno7.ht/politique/page/6/" rel="nofollow">https://www.juno7.ht/politique/page/6/</a>)<br>
2021-05-05 03:10:56 [scrapy.statscollectors] INFO: Dumping Scrapy stats:<br>
{'downloader/request_bytes': 5387,<br>
'downloader/request_count': 15,<br>
'downloader/request_method_count/GET': 15,<br>
'downloader/response_bytes': 531099,<br>
'downloader/response_count': 15,<br>
'downloader/response_status_count/200': 15,<br>
'dupefilter/filtered': 198,<br>
'elapsed_time_seconds': 14.06602,<br>
'finish_reason': 'shutdown',<br>
'finish_time': datetime.datetime(2021, 5, 5, 7, 10, 56, 112962),<br>
'log_count/DEBUG': 16,<br>
'log_count/INFO': 11,<br>
'request_depth_max': 7,<br>
'response_received_count': 15,<br>
'robotstxt/request_count': 1,<br>
'robotstxt/response_count': 1,<br>
'robotstxt/response_status_count/200': 1,<br>
'scheduler/dequeued': 14,<br>
'scheduler/dequeued/memory': 14,<br>
'scheduler/enqueued': 18,<br>
'scheduler/enqueued/memory': 18,<br>
'start_time': datetime.datetime(2021, 5, 5, 7, 10, 42, 46942)}<br>
2021-05-05 03:10:56 [scrapy.core.engine] INFO: Spider closed (shutdown)</p>