YaraAlnaggar:

I am expecting the inference of the two models to happen in a parallel since cuda is running asynchronously.


You’re not using torch.cuda.Stream, this means that single serialized queue of cuda operations is used. So, even if your first model is non-blocking (memory copying is blocking), async mode of cuda by itself only allows you to enqueue ops from second model earlier.