Hello Victor!



 victorc25:

Is it possible to multiply a trained model by a single scalar that is trainable and backpropagate the loss to that scalar?


Yes.  In general you can train additional parameters (e.g., things that
aren’t weights in Linears) by adding them as Parameter 10s to your
model (which should be a torch.nn.Module) or to your Optimizer.
Pytorch will do the rest.


I want to do this for two networks, so the idea is to train those two scalars to find how to ideally combine the two models’ weights (not the outputs).


I’m not sure exactly what you have in mind here.  But here is one idea
that might be relevant to your use case:
You have model A and model B already trained.  Freeze their layers
(.requires_grad = False).  Create an Optimizer that contains your
weight parameter as a single trainable parameter:
weight_param = torch.FloatTensor ([1.234])
optim = torch.optim.SGD ([weight_param])

(where 1.234 is your initial value of weight_param).
Then do something like:
xA = layer1A (input)
xB = layer1B (input)
x = weight_param * xA + (1.0 - weight_param) * xB
x = activation1 (x)
xA = layer1A (x)
xB = layer1B (x)
x = weight_param * xA + (1.0 - weight_param) * xB
x = activation2 (x)
...
loss = criterion (x, target)
optim.zero_grad()
loss.backward()
optim.step()

(This code can be rearranged in various ways, but this is the basic
idea.)
Is this the line along which you were thinking?
Best.
K. Frank