So you mean to say that x and y from the next batch will not be loaded to GPU while model inference is happening.
It’s just that with torch.cuda.synchronize() python will wait at this line for model inference to complete and then go and execute st = time.time(), while without it, it goes to x.cuda() and waits there.
If that’s true, is there a way to load the next batch while model inference is going on? That’s what I’m trying to do.