Solved by ptrblck in post #2 
              
              
                You can use torch.cuda.Stream to create custom streams. Check the docs carefully (as well as the streams part here) as I would consider it an advanced mechanism since you can easily create race conditions in your code.