I did do the running_mean and running_var for bn layers, the result is closer if i propagate the same tensor through all branches but still incorrect.
Would it be easier to make 3 instances of the same model, load the state dictionary (which holds trained branch 0 ) and somehow merge the branch 0-s together and rename them 1/2? Altought not sure how could I do that either