django-celery infrastructure over multiple servers, broker is redis
What will strongly simplify your processing is some shared storage, accessible from all cooperating servers. With such design, you may distribute the work among more servers without worrying on which server will be next processing step done.
Using AWS S3 (or similar) cloud storage
If you can use some cloud storage, like AWS S3, use that.
In case you have your servers running at AWS too, you do not pay for traffic within the same region, and transfers are quite fast.
Main advantage is, your data are available from all the servers under the same bucket/key name, so you do not have to bother about who is processing which file, as all have shared storage on S3.
note: If you need to get rid of old files, you may even set up some policy file on give bucket, e.g. to delete files older than 1 day or 1 week.
Using other types of shared storage
There are more options

Samba
central file server
FTP
Google storage (very similar to AWS S3)
Swift (from OpenStack)
etc.

For small files you could even use Redis, but such solutions are for good reasons rather rare.
Currently we have everything setup on single cloud server, that includes:

Database server
Apache
Celery
redis to serve as a broker for celery and for some other tasks
etc

Now we are thinking to break apart the main components to separate servers e.g. separate database server, separate storage for media files, web servers behind load balancers. The reason is to not to pay for one heavy server and use load balancers to create servers on demand to reduce cost and improve overall speed.
I am really confused about celery only, have anyone ever used celery on multiple production servers behind load balancers? Any guidance would be appreciated.
Consider one small use case which is currently how it is been done on single server (confusion is that how that can be done when we use multiple servers):

User uploads a abc.pptx file->reference is stored in database->stored on server disk
A task (convert document to pdf) is created and goes in redis (broker) queue
celery which is running on same server picks the task from queue

Read the file, convert it to pdf using software called docsplit
create a folder on server disk (which will be used as static content later on) puts pdf file and its thumbnail and plain text and the original file


Considering the above use case, how can you setup up multiple web servers which can perform the same functionality?