The first CUDA call will initialize the CUDA context, which will use some memory on your device (depending on the CUDA version, used GPU etc.).
You could check the memory usage in PyTorch using print(torch.cuda.memory_allocated()).