I have two trained neural networks (NNs) that I want to combine to create a new neural network (with the same structure) but whose weights are a combination of the previous two neural networks’ weights.
The two NNs have an accuracy of ~97%, but when I combine them I obtain a value of around 47%. The problem is not that one (combining non-linear things not always works), but instead that when I run the code below (I have it in a Jupyter cell) several times, the accuracy increases to around ~97%.
Any ideas what is happening?
The code that I have is this:
# The interpolation parameter    
beta = 0.5

# Get the networks' parametes
params1 = net1.named_parameters()
params2 = net2.named_parameters()

dict_params = dict(params2)

# Do the linear combination of the 2 NNs
for name1, param1 in params1:
    if name1 in dict_params:
        dict_params[name1].data.copy_(beta*param1.data + (1.0 - beta)*dict_params[name1].data)

# Create a new NN with the parameters
net_combined = Net().to(device)
net_combined.load_state_dict(dict_params)