ptrblck:

In that case you could get the state_dicts from each model, average all parameters (or use another reduction instead of the mean), and reload the state_dict to a single model.


I’ve done that, but now what I want to try to do is to optimize the linear combination of the models. I’ve been able to do something like that manually (very inefficient), but I wanted to see if there’s a way to do it using Pytorch. The struggle I have is finding a way to have the vector that represents the coefficient for each model in the combination inside the path of the backprop, so I can use Pytorch to tune those coefficients. I’m at a loss there, I don’t know if it’s even possible.



 ptrblck:

Given that each model might have converged to another local minimum, I don’t think that e.g. the average of all parameters representing local minima will give you another minimum.


And you are absolutely correct. I think there’s a possibility for it to work, given my previous manual tests, but that’s why I want to keep testing