Celery alone is precisely made to do what you need - no need to reinvent the wheel.
I have thousands of files stored in MongoDB which I need to fetch and process.
Processing consists of a few steps which should be done sequentially. The whole process takes around ~2 mins per file from start to end.
My question is how to do that as fast as possible while being scalable in future? Should I do it in pure python or should I maybe use Airflow + Celery (or even Celery by itself)? Are there any other ways/suggestions I could give a try?
Any suggestion is appreciated.
Thanks in advance!
Processing thousands of files in parallel