In case you are using the GPU, did you synchronize the code before starting and stopping the timers?
If so, are you seeing the slowdown only in the first iterations (this would be expected, if e.g. cudnn.benchmark is used or in case the JIT is optimizing the graph)?