Stack Exchange
How to scrape and infinity scrolling page?
I was trying to scrape the men's coats and jackets category in next.co.uk and I realized that the page has the infinity scrolling page
# -*- coding: utf-8 -*-
import scrapy
from ..items import NextItem

class NewoneSpider(scrapy.Spider):
    name = 'newOne'
    allowed_domains = ['www.next.co.uk']
    start_urls = [
        'https://www.next.co.uk/shop/gender-newbornboys-gender-newbornunisex-gender-olderboys-gender-youngerboys-productaffiliation-coatsandjackets-0'
        ]

    def parse(self, response):
        items = NextItem();
        global productCategory
        global productSubCategory
        products = response.css('.Details')
        currentUrl = response.request.url

        for product in products:
            productCategory = 'Furniture'
            productSubCategory = 'living Room'
            productCountry = 'uk'
            productSeller = 'John Lewis'
            productLink = product.css('.TitleText::attr(href)').extract_first()
            productTitle = product.css('.Desc::text').extract_first()
            productImage = product.css('.Image img::attr(src)').extract_first()
            productSalePrice = product.css('.Price a::text').extract_first()

            items['productCategory'] = productCategory
            items['productSubCategory'] = productSubCategory
            items['productCountry'] = productCountry
            items['productSeller'] = productSeller
            items['productLink'] = productLink
            items['productTitle'] = productTitle
            items['productImage'] = productImage
            items['productSalePrice'] = productSalePrice

            yield items

I was able to scrape 28 items and I can see more than that on on the website which has infinite scrolling implementation.

When you scroll down the page sends XHR call to the server and asks for more data.
Example:
https://www.next.co.uk/shop/gender-newbornboys-gender-newbornunisex-gender-olderboys-gender-youngerboys-productaffiliation-coatsandjackets/isort-score-minprice-0-maxprice-30000-srt-24

Each request is almost the same but the last element in the url is growing up by 24:

srt-24
srt-48 
srt-72

Now that you know how the "infinity" works, you can try and simulate it with code.
Example:
import requests

URL_TEMPLATE = 'https://www.next.co.uk/shop/gender-newbornboys-gender-newbornunisex-gender-olderboys-gender-youngerboys-productaffiliation-coatsandjackets/isort-score-minprice-0-maxprice-30000-srt-{}'

for step in range(24, 240, 24):
    r = requests.get(URL_TEMPLATE.format(step))
    if r.status_code == 200:
        # TODO We have the data - lets parse it
        pass


still scraping the normal amount of items

@fafoworatobi I am not sure I understand your comment.

it start from page two items and scrape only page two items

@fafoworatobi Did you look at my example code? Can you share your updated code?

it's working
i saw your example right now
i have been able to scrape 216 items from that category

wat if i have more than one link i want to scrape how would the code look like

if i have two links i want to scrape from like
https://www.next.co.uk/shop/gender-newbornboys-gender-newbornunisex-gender-olderboys-gender-youngerboys-productaffiliation-coatsandjackets-0
and
https://www2.next.co.uk/shop/gender-men-productaffiliation-coatsandjackets-0
how would the code be like
