I have k classes, for each of which I have trained a model. I would like to create a new model whose architecture is identical and whose parameters are the weighted average of the corresponding parameters of each of my k models according to some specified distribution p (e.g. [0.1, 0.2, 0.7] for k=3).
My goal is to construct such a model, evaluate it on data, and then have gradient propagate back to the original models (the intermediate averaged model does not matter to me). Would someone be able to help me understand the best way to do this?