Yes, your statement is correct.
The first iteration of the DataLoader would be slower since all workers are loading a complete batch. If the data loading is not a bottleneck, the data loading time should decrease towards zero.
To accelerate the model, you could use e.g. mixed-precision training and check if this would yield a speedup.
Also, torch.backends.cudnn.benchmark = True would enable cudnn to profile the kernels for each new input shape and could accelerate the training.