If you are using the GPU, did you synchronize the code before starting and stopping the timer?
This would be necessary, since CUDA operations are asynchronous and thus non-blocking.
If so, what does small/large refer to? Is it the depth of the model or also the number of parameters?