Hello Frank! Thank you very much for your reply!
So, the overall idea is that I want to combine two models that were fine-tuned independently into a single one by interpolating their weights, but I would like to do this using the PyTorch framework (and, most importantly, the loss functions and autograd).
Originally, I was thinking about a single “coefficient” for each model that could just multiply all of the model’s trained weight, but I think I understand your code and it makes sense to me!
If I understand correctly, it would be adding an activation after every layer of the original models that combines the corresponding layer of the two original models, is that right? What could be a good activation to test with in this case?
This sounds great!