Your system might be busy with CPU operations (such as data loading) and the GPU might thus need to wait. You could profile the code and check, if e.g. the data loading is indeed using the CPU resources until the GPU can execute the forward pass (or alternatively remove all data loading, create random tensors on the GPU, execute the forward pass, and check nvidia-smi again).