Hi Victor!



 victorc25:

it would be adding an activation after every layer of the original models that combines the corresponding layer of the two original models, is that right? What could be a good activation to test with in this case?


I was imagining that you had two models with the same architecture,
that is, that all of the layers and activations match up.  The only
difference is that they have been trained (or fine tuned) differently,
so that the trained weights differ.
In this case, model A’s activation1 and model B’s activation1
are the same, so I was thinking that you would simply use
activation1 after interpolating the results of layer1A and layer1B.
(Note, the result of interpolating these two results is the same as
what you would get by interpolating the linear layers layer1A and
layer1B, and then applying the interpolated layer to the upstream
input.)
(As an aside, I’m not convinced your scheme will work.  If model A
and model B are trained independently to do the same thing, the
values of the internal “hidden” neurons may still be very different,
and the layer weights may be very different, so averaging them might
not make sense.  If one model was first trained, and then fine tuned
separately to get models A and B, the weights in A and B might line
up sensibly, but I could imagine that averaging the models together
could simply serve to average away the results of the fine tunings.)
Good luck.
K. Frank