There is some solutions to run a SPA in real browser even in a headless environment<p>The trick is to emulate x11 with xvfb and control it with selenium web driver.<p>Phantom isn&#x27;t the only choice, just the one most people talk about<p>As for non js heavy website, it&#x27;s fairly trivial to find a library that will parse the dom for you, pretty every language have one