Web crawlers and Google App Engine Hosted applications
App Engine code only runs in response to HTTP requests, so you can't run a persistent crawler in the background. With the upcoming release of scheduled tasks, you could write a crawler that uses that functionality, but it would be less than ideal.
Is it impossible to run a web crawler on GAE along side with my app considering the I am running the free startup version?