Thought you might want to know that the second set of tests are done and posted on Github. A third set of trials with smaller model size is in the works. I’ve also added some information in the main readme file explaining what I’m doing and the motivation behind it(this is not just arbitrarily copying parameter values) Also uploaded a very simple spreadsheet example proving how the splitting of neurons through this method keeps the outputs unchanged through mat mul. This method only provides the model additional neurons to tweak, targetting only the neurons having the highest average loss between epochs.
Now I am trying to automate passing the optimizer references between epochs, without having to recreate the optimizer. I’ve updated the optimizer.state tensors. But it seems there is more to it than that. Just like when I updated the model attributes via,
def update_features(model, attr, targ, ins, outs):
    setattr(model, attr, type(targ)(ins, outs))

def replace_network(model, new_params):
    c=0
    for attr in dir(model):
        targ=getattr(model,attr)
        if type(targ)==nn.Linear:
            print(new_params[c].size()[1],new_params[c].size()[0])
            update_features(model,attr,targ,new_params[c].size()[1],new_params[c].size()[0])
            c=c+2
    with torch.no_grad():
        for i, p in enumerate(model.parameters()):
            p.copy_(new_params[i])

replace_network(net,new_params)

Is there a similar method for updating the optimizer’s group_params?