Hello Frank!



 KFrank:

I was imagining that you had two models with the same architecture,
that is, that all of the layers and activations match up. The only
difference is that they have been trained (or fine tuned) differently,
so that the trained weights differ.


Yes, that’s exactly the case!



 KFrank:

In this case, model A’s activation1 and model B’s activation1
are the same, so I was thinking that you would simply use
activation1 after interpolating the results of layer1A and layer1B .
(Note, the result of interpolating these two results is the same as
what you would get by interpolating the linear layers layer1A and
layer1B , and then applying the interpolated layer to the upstream
input.)


I understand now! Thanks for explaining it to me, it makes sense now. I’ll try that today to see how it goes, but it really sounds like what I wanted to test.



 KFrank:

(As an aside, I’m not convinced your scheme will work. If model A
and model B are trained independently to do the same thing, the
values of the internal “hidden” neurons may still be very different,
and the layer weights may be very different, so averaging them might
not make sense. If one model was first trained, and then fine tuned
separately to get models A and B, the weights in A and B might line
up sensibly, but I could imagine that averaging the models together
could simply serve to average away the results of the fine tunings.)


And you are exactly correct. The models line up (the filters are correlated) and I’ve tested doing this combination manually and it works, but I wanted to test using Pytorch’s automatic differentiation to optimize the search automatically and see how it behaves using some custom loss functions as well, so I think your suggestion will work perfectly!



 KFrank:

Good luck.


Thanks again!!