Web Scraping with Python
I think the overall software architecture approach of this post is fundamentally backwards. Given how much of the web is rendered client-side these days you need to start out with a headless option. Headless means that you fire up a true browser and then automate the actions that you need to perform. It&#x27;s indistinguishable from a real person using a browser. If you try to use urllib3 on a webpage that does heavy client-side rendering then you&#x27;re going to get incomplete HTML returned from the server (because the website intends to use JavaScript to complete the rendering of the page). On the more rare occasions when you are dealing with static HTML (i.e. there is no rendering on the client; the HTML returned from the server is the complete content) then you can use something like urllib3.<p>Re: which headless library to use this post mentions Selenium which was one of the first headless libs but from what I&#x27;ve heard probably not the best (in terms of developer experience or reliability or robustness) but that&#x27;s only hearsay... I&#x27;ve never used Selenium myself. Playwright seems like the best option in town if you want to use Python. Built by the former Chrome DevTools team (meaning those people really know how browser internals work). <a href="https:&#x2F;&#x2F;playwright.dev&#x2F;python&#x2F;docs&#x2F;intro" rel="nofollow">https:&#x2F;&#x2F;playwright.dev&#x2F;python&#x2F;docs&#x2F;intro</a>