Are you putting the net/model in cuda? Unlike Tensorflow, here you have to specify that you want to use GPU.
For small models there isn’t much difference, but for big models there difference is quite big.