I will check the synchronization, thanks! I have torch.cuda.synchronize() at the beginning of the script. Is this what you refer to?
Even though the ‘time per iter’ might not be accurate here, it also takes a much longer time to train over the same number of epochs with the same batch size, the same dataset.
By the size of the model, I meant to be the number of parameters.
Thanks again!