Post crawl stats collection
Try this command to save progress of spider:
nohup scrapy crawl spider > spider_progress &
See only tail of the text file which contains stats of spider: 
cat spider_progress | tail
I would like to collect and parse stats after my scrapy crawl has finished. I know it dumps the stats but I would like to ideally have a method/hook a method that runs when the scrape has finished which can then collect the stats (as I will use them to decide if another scrape is needed automatically).
Please don't suggest using some sort of external script, I want it all contained to one project please.