title: Fine grained Kernel scheduling with MPS
body: <p>I am working on using <a href="https://docs.nvidia.com/deploy/mps/index.html" rel="nofollow noreferrer">NVIDIA CUDA Multi Process Service (MPS)</a> for running multiple TensorFlow inference jobs using the same GPU. For my use-case, when GPU is being shared by more than one processes, I sometimes need to prioritize execution of kernels of one process over the other. Is this supported?</p>
  <p>To explain the problem in more detail, consider an example in which we have two processes, p1 and p2 (each with just one kernel execution stream) sharing a GPU.</p>
  <p><em>Scenario: When there are one or more kernels in ready queue for both p1 and p2.</em></p>
  <p><strong>Default MPS behavior (My understanding):</strong></p>
  <p>If there is enough resources, execute multiple kernels at the same time from both p1 and p2.</p>
  <p><strong>Desired behavior:</strong>
  Ability to decide based on priority if:</p>
  <ul>
  <li>Execute kernel of p1 first then p2.</li>
  <li>Execute kernel of p2 first then p1.</li>
  <li>Incase there is enough resources, execute multiple kernels at the same time from both p1 and p2.</li>
  </ul>
  <p>If this kind of customized scheduling is not supported, It will be great if someone can guide what code changes will be needed to make it work.</p>