Scraping websites - Online or offline data processing is better
Do it at the same time. It will probably only take a handful of lines of code. There's no reason to do the work of walking over the whole file twice.
I am scraping websites for a research project using Python Beautifulsoup.
I have scraped a few thousand records and put them in excel.
In essence, I want to extract a substring of text (e.g. "python" from a post-title "Introduction to python for dummies").
The post-title is scraped and stored in a cell in excel.
I want to extract "pyhon" and put it in another cell.
I need some advice if it was better to do the extraction while scraping OR do it offline in excel.
Since this is research project, there is no need for real time speed. i am looking at saving my effort.
Another related question is if python can be used to do the extraction in the offline mode - i.e. open excel, do the extraction , close excel.
Any help or advice is really appreciated.