The timing might be masked by another bottleneck in your code.
The training procedure on the GPU might be fast, but the overall training routine has to wait e.g. for the DataLoader to provide a new batch.