Since CUDA calls are asynchronous, you should synchronize all CUDA calls before starting and stopping the timer using torch.cuda.synchronize(). Currently you are most likely measuring the time to launch the kernels, not their real execution time.
Also, Variables are deprecated, so if you are using a newer PyTorch version (> 0.3.1) you can just remove the Variable wrappers.