How to evenly distribute tasks among nodes with Celery?
it's actually easy: you start one celery-instance per ec2-instance. set concurrency to the number of cores per ec2-instance.
now the tasks don't interfere and distribute nicely among you instances.
(the above assumes that your tasks are cpu bound)
I am using Celery with Django to manage a task que and using one (or more) small(single core) EC2 instances to process the task.
I have some considerations.

My task eats 100% CPU on a single core. - uses whatever CPU available but only in one core
If 2 tasks are in progress on the same core, each task will be slowed down by half.
I would like to start each task ASAP and not let it be que.

Now say I have 4 EC2 instances, i start celery with "-c 5" . i.e. 5 concurrent tasks per instance.
In this setup, if I have 4 new tasks, id like to ensure, each of them goes to different instance, rather than 4 going to same instance and each task fighting for CPU.
Similarly, if I have 8 tasks, id like each instance to get 2 tasks at a time, rather than 2 instances processing 4 tasks each.
Does celery already behave the way I described? If not then how can i make it behave as such?