Hi Frank!

I have concerns about combining weights like this from two
independently-trained models. (I have heard reports that this
works or can be made to work, but I’m skeptical.)

Can you send me these papers/reports, if you can find? Because I am doing this experiment for my thesis, and I just started with this.

The issue is that there is a lot of arbitrariness and redundancy
in the weights of a neural-network model. If I take two identical
models, but give them different (but equivalent) initializations,
and train them on the same training data (but probably batched
up into different (but equivalent) random batches), there is no
reason for “weight-17” in model A to have the same value as
“weight-17” in model B.

Yes, I know, but my supervisor assigned me this work, and we want to obtain these bad results. My supervisor has several ideas on how to try to improve later.
For the technical solution: I have modelMNIST, and mondelSVHN
You mean, I have to do this?
with torch.no_grad():
    model3.weight.copy_ (modelMNIST.block_1[0].weight + modelSVHN.block_1[0].weight

Doing it for all the blocks (I have 4 blocks + 1 classifier) and for each layer of the blocks?