I am scraping data of football player statistics from the web using python and Beautiful Soup. I will be scraping from multiple sources, and each source will have a variety of variables about each player which include strings, integers, and booleans. For example player name, position drafted, pro bowl pick (y/n).
Eventually I would like to put this data into a data mining tool or an analysis tool in order to find trends. This will need to be searchable and I will need to be able to add data to a player's info when I am scraping from a new source in a different order.
What techniques should I use to store the data so that I will best be able to add too it and the analyze it later?
Use a layered approach: downloading, parsing, storage, analysis.
Separate the layers. Most importantly, don't just download data and then store it in the final parsed format. You will inevitably realise you missed something and need to scrape it all over again. Use something like requests + requests_cache (I found that extending requests_cache.backends.BaseCache and storing it on the filesystem is more convenient examining scraped html than the default sqlite storage backend).
For parsing you're already using beautiful soup which works fine.
For storage & analysis use a database. Avoid the temptation to go with NoSQL -- as soon as you need to run aggregate queries you'll regret it.
Best way to store scraped data in Python for analysis