sorry, not sure if i mentioned that.
but the code i provided in the example is from my code where x is the result of a forward in large network. so a simple snippet wont work.
the forward needs to be long enough so the cuda kernels are launched but not finished, while the cpu has moved to next instructions such as print(t).
here is a full dummy example. in this example, because i synched after computing z, it is not the forward that is slow, it the the min/max op over a large tensor.
import time

import torch


class Module(torch.nn.Module):
    def forward(self) -> torch.Tensor:
        x = torch.rand(32, 256, 220, 220).cuda()
        s = torch.rand(32, 256, 220, 220).cuda()
        conv = torch.nn.Conv2d(256, 256, 3).cuda()
        z1 = torch.pow(x, 2)
        z2 = z1 / 1000.
        z3 = conv(z2) + conv(s)
        z3 = conv(torch.pow(z3, 2) + z3 * 2 / 100.)
        z4 = z3 / 100.

        return z4


if __name__ == '__main__':
    inst = Module().cuda()
    start_event = torch.cuda.Event(enable_timing=True)
    end_event = torch.cuda.Event(enable_timing=True)
    start_event.record()
    z = inst()
    end_event.record()
    torch.cuda.synchronize()
    elapsed_time_ms = start_event.elapsed_time(end_event)
    print('time : {}'.format(elapsed_time_ms))

    t = (z.min() - z.max()).to(torch.device("cpu"), non_blocking=True)
    print(t)
    time.sleep(2.)
    print(t)

output
time : 7206.33154296875
tensor(0., grad_fn=<CopyBackwards>)
tensor(-0.0099, grad_fn=<CopyBackwards>)

here is a simple snippet with large tensors:
import time

import torch


if __name__ == '__main__':
    seed = 0
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

    x = torch.rand(32, 256, 220, 220).cuda()

    t = (x.min() - x.max()).to(torch.device("cpu"), non_blocking=True)
    print(t)
    time.sleep(2.)
    print(t)


output:
tensor(0.)
tensor(-1.0000)

i used https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py to collect info. let me know if you need more. thanks
$ python collect_env.py
Collecting environment information...
PyTorch version: 1.9.0
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.10

Python version: 3.7.9 (default, Aug 31 2020, 12:42:55)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-4.15.0-122-generic-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 10.0.130
GPU models and configuration: 
GPU 0: Tesla P100-PCIE-16GB
GPU 1: Tesla P100-PCIE-16GB

Nvidia driver version: 455.32.00
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.2
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] efficientnet-pytorch==0.7.0
[pip3] numpy==1.20.1
[pip3] torch==1.9.0
[pip3] torchvision==0.10.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] efficientnet-pytorch      0.7.0                    pypi_0    pypi
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.3.0           h06a4308_520  
[conda] numpy                     1.20.1                   pypi_0    pypi
[conda] pytorch                   1.9.0           py3.7_cuda11.1_cudnn8.0.5_0    pytorch
[conda] torchvision               0.10.0               py37_cu111    pytorch