Hi,
Which time library are you using? Is it cuda aware and doing the proper synchronization (via torch.cuda.synchronize() for example)?
Also your code creates the Tensors on cpu then move them to the gpu, you can do the following to avoid this: torch.zeros(4, 1, 20, 20, 20, 32, device="cuda")
The problem with cudaMalloc/Free is that they need to access the GPU and so can be fairly slow. Allocating a Tensor should be faster because we use a custom allocator.