Caching large objects in a python Flask/Gevent web service
Can't you unpickle the files when the sever is instanciated, and then keep the unpickled data into the global namespace? This way, it'll be available for each requests, and as you're not planning to write anything in it, you do not have to fear any race conditions.
I am building a python based web service that provides natural language processing support to our main app API. Since it's so NLP heavy, it requires unpickling a few very large (50-300MB) corpus files from the disk before it can do any kind of analyses. 
How can I load these files into memory so that they are available to every request? I experimented with memcached and redis but they seem designed for much smaller objects. I have also been trying to use the Flask g object, but this only persists throughout one request.
Is there any way to do this while using a gevent (or other) server to allow concurrent connections? The corpora are completely read-only so there ought to be a safe way to expose the memory to multiple greenlets/threads/processes.
Thanks so much and sorry if it's a stupid question - I've been working with python for quite a while but I'm relatively new to web programming.