I need to scrape a url which has checkboxes in it. I wanna click some of the checkboxes and scrape and I wanna scrape again with someother checkboxes clicked. For instance;
I wanna click new and then scrape
and then I wanna scrape the same url with Used and Very Good clicked.
Is there a way to do this without making more than 1 request which is done for getting the url.
I guess html changes when you click one of the boxes since the listing will change when you refine the search. Any thoughts? Any suggestions?
Best,
Can
You are wrong.
Scrapy cannot manipulate real browser-like behavior.
From the image you linked, I saw you are scraping Amazon, so open that link in browser, and click on checkbox, you will notice the URL in browser will also change according to new filter set.
And then put that URL in scrapy code and do your scraping.
IF YOU WANT TO MANIPULATE REAL BROWSER-LIKE BEHAVIOR use Python Selenium or PhantomJS or CasperJS.
Scrapy - How to put a check into checkboxes in a url then scrape