Do you print the index in your custom dataset class? This might be caused by the dataloader preloading data, which actually speeds up your later batches overall…
Could you print the shape of the imgs tensor in the training loop for i, (imgs, targets, _, _) in enumerate(dataloader):? My guess is that the batch size here should be 4.