title: How to best perform Multiprocessing within requests with the python Tornado server?
body: <p>I am using the I/O non-blocking python server Tornado.  I have a class of <code>GET</code> requests which may take a significant amount of time to complete (think in the range of 5-10 seconds).  The problem is that Tornado blocks on these requests so that subsequent fast requests are held up until the slow request completes.</p>
  
  <p>I looked at: <a href="https://github.com/facebook/tornado/wiki/Threading-and-concurrency">https://github.com/facebook/tornado/wiki/Threading-and-concurrency</a> and came to the conclusion that I wanted some combination of #3 (other processes) and #4 (other threads).  #4 on its own had issues and I was unable to get reliable control back to the ioloop when there was another thread doing the "heavy_lifting". (I assume that this was due to the GIL and the fact that the heavy_lifting task has high CPU load and keeps pulling control away from the main ioloop, but thats a guess).</p>
  
  <p>So I have been prototyping how to solve this by doing "heavy lifting" tasks within these slow <code>GET</code> requests in a separate process and then place a callback back into the Tornado ioloop when the process is done to finish the request. This frees up the ioloop to handle other requests.</p>
  
  <p>I have created a simple example demonstrating a possible solution, but am curious to get feedback from the community on it.</p>
  
  <p><strong>My question is two-fold: How can this current approach be simplified?  What pitfalls potentially exist with it?</strong></p>
  
  <h2>The Approach</h2>
  
  <ol>
  <li><p>Utilize Tornado's builtin <code>asynchronous</code> decorator which allows a request to stay open and for the ioloop to continue.</p></li>
  <li><p>Spawn a separate process for "heavy lifting" tasks using python's <code>multiprocessing</code> module.  I first attempted to use the <code>threading</code> module but was unable to get any reliable relinquishing of control back to the ioloop. It also appears that <code>mutliprocessing</code> would also take advantage of multicores.</p></li>
  <li><p>Start a 'watcher' thread in the main ioloop process using the <code>threading</code> module who's job it is to watch a <code>multiprocessing.Queue</code> for the results of the "heavy lifting" task when it completes.  This was needed because I needed a way to know that the heavy_lifting task had completed while being able to still notify the ioloop that this request was now finished.</p></li>
  <li><p>Be sure that the 'watcher' thread relinquishes control to the main ioloop loop often with <code>time.sleep(0)</code> calls so that other requests continue to get readily processed.</p></li>
  <li><p>When there is a result in the queue then add a callback from the "watcher" thread using <code>tornado.ioloop.IOLoop.instance().add_callback()</code> which is documented to be the only safe way to call ioloop instances from other threads.</p></li>
  <li><p>Be sure to then call <code>finish()</code> in the callback to complete the request and hand over a reply.</p></li>
  </ol>
  
  <p>Below is some sample code showing this approach.  <code>multi_tornado.py</code> is the server implementing the above outline and <code>call_multi.py</code> is a sample script that calls the server in two different ways to test the server.  Both tests call the server with 3 slow <code>GET</code> requests followed by 20 fast <code>GET</code> requests.  The results are shown for both running with and without the threading turned on.</p>
  
  <p>In the case of running it with "no threading" the 3 slow requests block (each taking a little over a second to complete).  A few of the 20 fast requests squeeze through in between some of the slow requests within the ioloop (not totally sure how that occurs - but could be an artifact that I am running both the server and client test script on the same machine).  The point here being that all of the fast requests are held up to varying degrees.</p>
  
  <p>In the case of running it with threading enabled the 20 fast requests all complete first immediately and the three slow requests complete at about the same time afterwards as they have each been running in parallel.  This is the desired behavior. The three slow requests take 2.5 seconds to complete in parallel - whereas in the non threaded case the three slow requests take about 3.5 seconds in total.  So there is about 35% speed up overall (I assume due to multicore sharing).  But more importantly - the fast requests were immediately handled in leu of the slow ones.</p>
  
  <p>I do not have a lot experience with multithreaded programming - so while this seemingly works here I am curious to learn:</p>
  
  <p><strong>Is there a simpler way to accomplish this?  What monster's may lurk within this approach?</strong>   </p>
  
  <p>(Note:  A future tradeoff may be to just run more instances of Tornado with a reverse proxy like nginx doing load balancing.  No matter what I will be running multiple instances with a load balancer - but I am concerned about just throwing hardware at this problem since it seems that the hardware is so directly coupled to the problem in terms of the blocking.)</p>
  
  <h2>Sample Code</h2>
  
  <p><strong><code>multi_tornado.py</code></strong> (sample server):</p>
  
  <pre><code>import time
  import threading
  import multiprocessing
  import math
  
  from tornado.web import RequestHandler, Application, asynchronous
  from tornado.ioloop import IOLoop
  
  
  # run in some other process - put result in q
  def heavy_lifting(q):
      t0 = time.time()
      for k in range(2000):
          math.factorial(k)
  
      t = time.time()
      q.put(t - t0)  # report time to compute in queue
  
  
  class FastHandler(RequestHandler):
      def get(self):
          res = 'fast result ' + self.get_argument('id')
          print res
          self.write(res)
          self.flush()
  
  
  class MultiThreadedHandler(RequestHandler):
      # Note:  This handler can be called with threaded = True or False
      def initialize(self, threaded=True):
          self._threaded = threaded
          self._q = multiprocessing.Queue()
  
      def start_process(self, worker, callback):
          # method to start process and watcher thread
          self._callback = callback
  
          if self._threaded:
              # launch process
              multiprocessing.Process(target=worker, args=(self._q,)).start()
  
              # start watching for process to finish
              threading.Thread(target=self._watcher).start()
  
          else:
              # threaded = False just call directly and block
              worker(self._q)
              self._watcher()
  
      def _watcher(self):
          # watches the queue for process result
          while self._q.empty():
              time.sleep(0)  # relinquish control if not ready
  
          # put callback back into the ioloop so we can finish request
          response = self._q.get(False)
          IOLoop.instance().add_callback(lambda: self._callback(response))
  
  
  class SlowHandler(MultiThreadedHandler):
      @asynchronous
      def get(self):
          # start a thread to watch for
          self.start_process(heavy_lifting, self._on_response)
  
      def _on_response(self, delta):
          _id = self.get_argument('id')
          res = 'slow result {} &lt;--- {:0.3f} s'.format(_id, delta)
          print res
          self.write(res)
          self.flush()
          self.finish()   # be sure to finish request
  
  
  application = Application([
      (r"/fast", FastHandler),
      (r"/slow", SlowHandler, dict(threaded=False)),
      (r"/slow_threaded", SlowHandler, dict(threaded=True)),
  ])
  
  
  if __name__ == "__main__":
      application.listen(8888)
      IOLoop.instance().start()
  </code></pre>
  
  <p><strong><code>call_multi.py</code></strong> (client tester):</p>
  
  <pre><code>import sys
  from tornado.ioloop import IOLoop
  from tornado import httpclient
  
  
  def run(slow):
      def show_response(res):
          print res.body
  
      # make 3 "slow" requests on server
      requests = []
      for k in xrange(3):
          uri = 'http://localhost:8888/{}?id={}'
          requests.append(uri.format(slow, str(k + 1)))
  
      # followed by 20 "fast" requests
      for k in xrange(20):
          uri = 'http://localhost:8888/fast?id={}'
          requests.append(uri.format(k + 1))
  
      # show results as they return
      http_client = httpclient.AsyncHTTPClient()
  
      print 'Scheduling Get Requests:'
      print '------------------------'
      for req in requests:
          print req
          http_client.fetch(req, show_response)
  
      # execute requests on server
      print '\nStart sending requests....'
      IOLoop.instance().start()
  
  if __name__ == '__main__':
      scenario = sys.argv[1]
  
      if scenario == 'slow' or scenario == 'slow_threaded':
          run(scenario)
  </code></pre>
  
  <h2>Test Results</h2>
  
  <p>By running <code>python call_multi.py slow</code>  (the blocking behavior):</p>
  
  <pre><code>Scheduling Get Requests:
  ------------------------
  http://localhost:8888/slow?id=1
  http://localhost:8888/slow?id=2
  http://localhost:8888/slow?id=3
  http://localhost:8888/fast?id=1
  http://localhost:8888/fast?id=2
  http://localhost:8888/fast?id=3
  http://localhost:8888/fast?id=4
  http://localhost:8888/fast?id=5
  http://localhost:8888/fast?id=6
  http://localhost:8888/fast?id=7
  http://localhost:8888/fast?id=8
  http://localhost:8888/fast?id=9
  http://localhost:8888/fast?id=10
  http://localhost:8888/fast?id=11
  http://localhost:8888/fast?id=12
  http://localhost:8888/fast?id=13
  http://localhost:8888/fast?id=14
  http://localhost:8888/fast?id=15
  http://localhost:8888/fast?id=16
  http://localhost:8888/fast?id=17
  http://localhost:8888/fast?id=18
  http://localhost:8888/fast?id=19
  http://localhost:8888/fast?id=20
  
  Start sending requests....
  slow result 1 &lt;--- 1.338 s
  fast result 1
  fast result 2
  fast result 3
  fast result 4
  fast result 5
  fast result 6
  fast result 7
  slow result 2 &lt;--- 1.169 s
  slow result 3 &lt;--- 1.130 s
  fast result 8
  fast result 9
  fast result 10
  fast result 11
  fast result 13
  fast result 12
  fast result 14
  fast result 15
  fast result 16
  fast result 18
  fast result 17
  fast result 19
  fast result 20
  </code></pre>
  
  <p>By running <code>python call_multi.py slow_threaded</code>  (the desired behavior):</p>
  
  <pre><code>Scheduling Get Requests:
  ------------------------
  http://localhost:8888/slow_threaded?id=1
  http://localhost:8888/slow_threaded?id=2
  http://localhost:8888/slow_threaded?id=3
  http://localhost:8888/fast?id=1
  http://localhost:8888/fast?id=2
  http://localhost:8888/fast?id=3
  http://localhost:8888/fast?id=4
  http://localhost:8888/fast?id=5
  http://localhost:8888/fast?id=6
  http://localhost:8888/fast?id=7
  http://localhost:8888/fast?id=8
  http://localhost:8888/fast?id=9
  http://localhost:8888/fast?id=10
  http://localhost:8888/fast?id=11
  http://localhost:8888/fast?id=12
  http://localhost:8888/fast?id=13
  http://localhost:8888/fast?id=14
  http://localhost:8888/fast?id=15
  http://localhost:8888/fast?id=16
  http://localhost:8888/fast?id=17
  http://localhost:8888/fast?id=18
  http://localhost:8888/fast?id=19
  http://localhost:8888/fast?id=20
  
  Start sending requests....
  fast result 1
  fast result 2
  fast result 3
  fast result 4
  fast result 5
  fast result 6
  fast result 7
  fast result 8
  fast result 9
  fast result 10
  fast result 11
  fast result 12
  fast result 13
  fast result 14
  fast result 15
  fast result 19
  fast result 20
  fast result 17
  fast result 16
  fast result 18
  slow result 2 &lt;--- 2.485 s
  slow result 3 &lt;--- 2.491 s
  slow result 1 &lt;--- 2.517 s
  </code></pre>