Question: GPU operations are not asynchronous in my case.
Description:
I run something like
t = time.time()
loss = model(x)
loss.backward()
cost = time.time() - t
but I got almost the same result with/without torch.cuda.synchronize().
I have called .cuda() for model.(the model is on gpu)
There should be no gpu-cpu transfer(i.e. .cpu() or .gpu()) in model’s forward() method
It seems that GPU operations are not asynchronous in my case.
Why?
Or how can I check if I mistakely sync during model’s forward() method?