When pytorch gets its first CUDA call, it needs to initialize CUDA, which is probably why the first one you see has additional overhead.
If you want to benchmark it more accurately you have to make sure that you initialize CUDA first before beginning timing. Something like x = torch.randn(1).cuda() or even just torch.cuda.init().