How can I load all items on a web page when using an HTML parser?
The Web browser itself can obviously load more items, and a web browser is just a program, so yes, it's possible to write a program to do it.
But you'd essentially be writing a mini web browser, which is a lot of work.
Does the website itself offer a way to load more items at once?  i.e. some websites offer a link or a dropdown menu in the corner of the page that allows you to increase the maximum number of items shown.
I'm trying to get a list of all the items available for purchase on a certain web page. However, the web page only loads 12 items at a time until the user scrolls down and then 12 more are loaded. Is there a way, in C# or Python, using any open source libraries, to be able to "see" all of the items available without physically going to the page and scrolling down?
Using Chrome's developer tools, I can only "see" the 12 items in the HTML window until I physically scroll down on the web page and it loads more.
NOTE: I'm relatively new at C#/Python web scraping, so I appreciate any in depth answers!
EDIT: If I were to use something like Selenium, would it be possible to load everything programatically? If so, how?