there might not be mistakes in your code.
If you run a very small model then it will have same speed on CPU and GPU.
If you run large model, you might see speedup.
You can look at our examples for GAN code which is fast: https://github.com/pytorch/examples/ 246