vkverma:

M=a1 M1.parameters()+a2 M2.parameters()


This line suggests that you want to do parameter averaging with replicated networks. That won’t work as network re-runs converge to distinct local minima.
OTOH, network output averaging (ensembles) can work, but you normally need a staged training for that. I.e. combination weights should be trained with independently pre-trained contributing models.