How would I scrape a domain to find all web pages and content?
For example: www.example.com, www.example.com/index.html, www.example.com/about/index.html and so on..
I would like to do this in Python and preferable with Beautiful Soup if possible..
You can't. Pages not only can pages be dynamically generated based on backend database data and search queries or other input that your program supplies to the website, but there is a nearly infinite list of possible pages, and the only way to know which ones exist is to test and see.
The closest you can get is to scrape a website based on hyperlinks between pages in the page content itself.
How can I find (and scrape) all web pages on a given domain using Python?