In addition to what @glaringlee said, the CPU tensors must also be in pinned memory for the copy to be non-blocking.
torch::Tensor cpu_tensor = torch::randn({128, 3, 24, 24}, torch::kFloat).pin_memory();