Ask HN: What are the best tools for web scraping in 2022?
Unpopular opinion, but Bash/Shell Scripting. Seriously, it's probably the fastest way to get things done. For fetching, use cURL. Want to extract particular markup? Use pup[1]. Want to process csv? Use cskit[2]. Or JSON? Use jq[3]. Want to use DB? Use psql. Once you get the hang of shell scripting, you can create simple scrapers by wiring up these utilities in a matter of minutes.<p>The only thing I wish was present was better support for RegExes. Bash and most unix tools don't support PCRE which can severely limiting. Plus, sometimes you want to process text as a whole vs line-by-line.<p>I would also recommend Python's sh[4] module if Shell scripting isn't your cup of tea. You get best of both worlds: faster dev work with Bash utils, and a saner syntax.<p>[1]: <a href="https://github.com/ericchiang/pup" rel="nofollow">https://github.com/ericchiang/pup</a><p>[2]: <a href="https://csvkit.readthedocs.io/en/latest/" rel="nofollow">https://csvkit.readthedocs.io/en/latest/</a><p>[3]: <a href="https://stedolan.github.io/jq/" rel="nofollow">https://stedolan.github.io/jq/</a><p>[4]: <a href="https://pypi.org/project/sh/" rel="nofollow">https://pypi.org/project/sh/</a>