Solved by ptrblck in post #2 
              
              
                If you are running the code on a GPU and call p1 and p2 after each other, these calls will be queued onto the device and executed asynchronously. Depending on the workload of p1, p2 might start while p1 is still executing. 
If you have multiple GPUs in your system (and don’t use data parallel), you …