Solved by ptrblck in post #2 
              
              
                To simply average the parameters of multiple models you could use this approach. However, based on your description you would like to calculate the gradients for the parameters in the original models, so using the averaged state_dict wouldn’t work. 
Instead I think the right approach would be to cre…