Scrapy using significant computer storage
You have a couple of options with your information given. Most notably, run the python script on your mac instead of the Bootcamp windows. If your storage is being eaten up, you can try writing to a .txt instead of .csv. Maybe try giving an example of what you are loading up / trying to do so I could replicate
I'm pretty new to python and scrapy. Right now I'm using a spider that is utilizing selenium to load JS pages. I'm running a windows computer through Bootcamp and I have about 10GB of storage left, however, when I run my scrapy spider, within 30 minutes of running it takes up almost half of that storage and I'm not sure why. I turned HTTP Cache off in settings.py but it still is eating up several GBs of storage.
I'm using the -o command in my cmd to save the output to a csv, but that's the only thing I can think of that would take up storage. Any advice would be greatly appreciated!