You can use torch.cuda.Stream to create custom streams. Check the docs carefully (as well as the streams part here 1) as I would consider it an advanced mechanism since you can easily create race conditions in your code.