@ptrblck Slightly off-topic: Is there any solid evidence that averaging the weights of different runs has any benefits?
In principle, both runs may converge two very different minima w.r.t. the the loss. It’s not obvious to my, why averaging the weights would be a good idea. Admittedly, this might depend what “trained exactly the same way” means, but even a different order of batches might effect this, wouldn’t it?