title: cudaDeviceSynchronize() waits to finish only in current CUDA context or in all contexts?
body: <p>I use <strong>CUDA 6.5</strong> and <strong>4 x GPUs Kepler</strong>.</p>
  
  <p>I use multithreading, CUDA runtime API and access to the CUDA contexts from different CPU threads (by using OpenMP - but it does not really matter).</p>
  
  <ol>
  <li><p>When I call <code>cudaDeviceSynchronize();</code> will it wait for kernel(s) to finish only in current CUDA context which selected by the latest call <code>cudaSetDevice()</code>, or in all CUDA contexts?</p></li>
  <li><p>If it will wait for kernel(s) to finish in all CUDA contexts, then it will wait in all CUDA contexts which used in current CPU thread (<em>in example CPU thread_0 will wait GPUs: 0 and 1</em>) or generally all CUDA contexts (<em>CPU thread_0 will wait GPUs: 0, 1, 2 and 3</em>)?</p></li>
  </ol>
  
  <p>Following code:</p>
  
  <pre><code>// For using OpenMP requires to set:
  // MSVS option: -Xcompiler "/openmp"
  // GCC option: –Xcompiler –fopenmp
  #include &lt;omp.h&gt;
  
  int main() {
  
      // execute two threads with different: omp_get_thread_num() = 0 and 1
      #pragma omp parallel num_threads(2)
      {
          int omp_threadId = omp_get_thread_num();
  
          // CPU thread 0
          if(omp_threadId == 0) {
  
              cudaSetDevice(0);
              kernel_0&lt;&lt;&lt;...&gt;&gt;&gt;(...);
              cudaSetDevice(1);
              kernel_1&lt;&lt;&lt;...&gt;&gt;&gt;(...);
  
              cudaDeviceSynchronize(); // what kernel&lt;&gt;() will wait?
  
          // CPU thread 1
          } else if(omp_threadId == 1) {
  
              cudaSetDevice(2);
              kernel_2&lt;&lt;&lt;...&gt;&gt;&gt;(...);
              cudaSetDevice(3);
              kernel_3&lt;&lt;&lt;...&gt;&gt;&gt;(...);
  
              cudaDeviceSynchronize(); // what kernel&lt;&gt;() will wait?
  
          }
      }
  
      return 0;
  }
  </code></pre>