GAE: scheduled import of big gzipped file from a third party site
My initial gut reaction (without knowing what's inside the gzipped file) is to do the processing somewhere else (AWS?) and then pushed the processed data to your GAE application in small bits.
I'm working on a Python web app that needs to import big (in terms of GAE limits) gzipped files from a third party site on regular basis. Think of rdf exports DMOZ project is producing on regular intervals.
This means daily fetching of a 500+ MB gzip file, gunzipping, parsing, processing and storing the results in GAE's datastore for later use.
What's the proper way to implement this functionality on GAE, having in mind the limits for maximum download, processing time, etc?