What is the fastest way to get scraped data from so many web pages?
Based on what I know about your app it would make sense to use memcache. It will be faster, and will automatically take care of things like expiring stale cache entries.
I need to scrap about 40 random webpages at the same time.These pages vary on each request.
I have used rpcs in python to fetch the urls and scraped the data using BeautifulSoup. It takes about 25 seconds to scrap all the data and display on the screen. 
To increase the speed i stored the data in appengine datastore so that each data is scraped only once and can be accessed from there quickly.
But the problem is-> as the size of the data increases in the datastore, it is taking too long to fetch the data from the datastore(more than the scraping).
Should i use memcache Or shift to mysql? Is mysql faster than gae-datastore?
Or is there any other better way to fetch the data as quickly as possible?