title: Using Celery for Realtime, Synchronous External API Querying with Gevent
body: <p>I'm working on a web application that will receive a request from a user and have to hit a number of external APIs to compose the answer to that request. This could be done directly from the main web thread using something like gevent to fan out the request.</p>
  
  <p>Alternatively, I was thinking, I could put incoming requests into a queue and use workers to distribute the load. The idea would be to try to keep it real time, while splitting up the requests amongst several workers. Each of these workers would be querying only one of the many external APIs. The response they receive would then go through a series transformations, be saved into a DB, be transformed to a common schema and saved in a common DB to finally be composed into one big response that would be returned through the web request. The web request is most likely going to be blocking all this time, with a user waiting, so keeping
  the queueing and dequeueing as fast as possible is important. </p>
  
  <p>The external API calls can easily be turned into individual tasks. I think the linking
  from one api task to a transformation to a DB saving task could be done using a chain, etc, and the final result combining all results returned to the web thread using a chord. </p>
  
  <p>Some questions: </p>
  
  <ul>
  <li>Can this (and should this) be done using celery?</li>
  <li>I'm using django. Should I try to use django-celery over plain celery?</li>
  <li>Each one of those tasks might spawn off other tasks - such as logging what just
  happened or other types of branching off. Is this possible?</li>
  <li>Could tasks be returning the data they get - i.e. potentially Kb of data through celery (redis as underlying in this case) or should they write to the DB, and just pass pointers to that data around?</li>
  <li>Each task is mostly I/O bound, and was initially just going to use gevent from the web thread to fan out the requests and skip the whole queuing design, but it turns out that it would be reused for a different component. Trying to keep the whole round trip through the Qs real time will probably require many workers making sure the queueus are mostly empty. Or is it? Would running the gevent worker pool help with this?</li>
  <li>Do I have to write gevent specific tasks or will using the gevent pool deal with network IO automagically?</li>
  <li>Is it possible to assign priority to certain tasks?</li>
  <li>What about keeping them in order?</li>
  <li>Should I skip celery and just use kombu?</li>
  <li>It seems like celery is geared more towards "tasks" that can be deferred and are
  not time sensitive. Am I nuts for trying to keep this real time?</li>
  <li>What other technologies should I look at?</li>
  </ul>
  
  <p>Update: Trying to hash this out a bit more. I did some reading on Kombu and it seems to be able to do what I'm thinking of, although at a much lower level than celery. Here is a diagram of what I had in mind. <img src="https://i.stack.imgur.com/TDwGA.png" alt="It&#39;s a simplified version, i.e. skipping the DB saving steps done by worker_2."></p>
  
  <p>What seems to be possible with raw queues as accessible with Kombu is the ability for a number of workers to subscribe to a broadcast message. The type and number does not need to be known by the publisher if using a queue. Can something similar be achieved using Celery? It seems like if you want to make a chord, you need to know at runtime what tasks are going to be involved in the chord, whereas in this scenario you can simply add listeners to the broadcast, and simply make sure they announce they are in the running to add responses to the final queue.</p>
  
  <p>Update 2: I see there is the <a href="http://docs.celeryproject.org/en/latest/userguide/routing.html#broadcast" rel="noreferrer">ability to broadcast</a> Can you combine this with a chord? In general, can you combine celery with raw kombu? This is starting to sound like a question about smoothies.</p>