title: Keras + Tensorflow: Prediction on multiple gpus
body: <p>I'm using Keras with tensorflow as backend.
  I have one compiled/trained model.</p>
  
  <p>My prediction loop is slow so I would like to find a way to parallelize the <code>predict_proba</code> calls to speed things up.
  I would like to take a list of batches (of data) and then per available gpu, run <code>model.predict_proba()</code> over a subset of those batches.<br>
  Essentially:</p>
  
  <pre><code>data = [ batch_0, batch_1, ... , batch_N ]
  on gpu_0 =&gt; return predict_proba(batch_0)
  on gpu_1 =&gt; return predict_proba(batch_1)
  ...
  on gpu_N =&gt; return predict_proba(batch_N) 
  </code></pre>
  
  <p>I know that it's possible in pure Tensorflow to assign ops to a given gpu (<a href="https://www.tensorflow.org/tutorials/using_gpu" rel="noreferrer">https://www.tensorflow.org/tutorials/using_gpu</a>).  However, I don't know how this translates to my situation since I've built/compiled/trained my model using Keras' api.</p>
  
  <p>I had thought that maybe I just needed to use python's multiprocessing module and start a process per gpu that would run <code>predict_proba(batch_n)</code>.  I know this is theoretically possible given another SO post of mine: <a href="https://stackoverflow.com/questions/42504669/keras-tensorflow-and-multiprocessing-in-python">Keras + Tensorflow and Multiprocessing in Python</a>.  However, this still leaves me with the dilemma of not knowing how to actually "choose" a gpu to operate the process on.</p>
  
  <p>My question boils down to: how does one parallelize prediction for one model in Keras across multiple gpus when using Tensorflow as Keras' backend?</p>
  
  <p>Additionally I am curious if similar parallelization for prediction is possible with only one gpu.   </p>
  
  <p>A high level description or code example would be greatly appreciated!</p>
  
  <p>Thanks!</p>