I notice that Pytorch has slow training process when I train the model for the first time. But the next time the speed is significantly improved. This makes me think that some caching is done after first training for faster inference or data loading in Pytorch.
One important thing I notice is that the starting loss of later training is much lower than the starting loss in the first training. So I wonder if Pytorch has done something with the weight initialization?
I hope somebody will explain to me this kind of phenomenon so I don’t have to worry about my training code.