That doesn’t sound like backward() issue. Make sure your use same inputs & targets as when separately training model2. If first epoch’s loss_2 is the same (with same parameters and inputs), investigate why optimizer does different things.