Is it feasible to unpickle large amounts of data in a Python web app that needs to be scalable?
Pickle files is good way to load data in python. By the way consider using the C implementation: cPickle.
If you want to scale, using pickle files. Ideally you want to look for a partition key, that fits for your data and your project needs.
Let's say for example you have historical data, instead of having a single file with all historical data, you can create a pickle file per date.
I am working on my first web application. I am doing it in Python with Flask, and I want to run a piece of Python code that takes an object, with Pickle, from a (binary) file of several MB which contains all the necessary data. I have been told that so using Pickle in a web app is not a good idea because of scalability; why?
Obviously, for a given purpose it's better to take just the necessary data. However, if I do this, with an Elasticsearch database and in the fastest way I can, the whole process takes about 100 times more time than if I take all the data at once from the binary; once the binary is unpickled, which will take at most one second, the computations are very fast, so I am wondering if I should use the binary file, and if so, how to do it in a scalable way.