Scheduling tasks with python/celery/any other task queue based on available RAM
Allocating tasks to workers based on available RAM is not possible. But, depending on your setup and exact requirements there are ways to work address this problem. Off the top of my head I can think of:

you run one worker (and nothing else) on a machine with loads of RAM
and have this worker process these memory-hungry tasks only
if you know how much memory you need for processing a task, you could
check if that memory is available as a first step in your task
(assuming that not much else runs on that box which could use up that
memory in between the check and the actual running of your
memory-hungry stuff) and if not, you can retry the task after a
certain time
Is there a way to get Celery to allocate tasks to workers based on available RAM? I am developing a web app which applies memory hungry algorithms to large in-memory data, is there a way to make this more robust using Celery to respect available RAM?