Yup the repo contains cuda implementation of a layer(lsoftmax) but I did not use the layer. Like I said, I run the code using the command python2 mnist.py --train --no-lsoftmax --gpu 0 --batch_size 256.
Hence it is using a normal fully connected layer in the last layer.
Also I tried removing the printing lines, and it does not make much speed difference, still 3s.