Thanks a lot, that’s great!
When I see it correctly though, there’s no update of inactive parameters, i.e.,  when I run the code in PyTorch 0.4, for the decoder 1 branch, I get
Dec2 grad: None
Diff in enc.weight
Diff in enc.bias
Diff in dec1.weight
Diff in dec1.bias

And vice versa, I get the following for the decoder 2 branch:
Diff in enc.weight
Diff in enc.bias
Diff in dec2.weight
Diff in dec2.bias

So, using something like
itertools.chain(model.enc.parameters(), model.dec1.parameters())
instead of the if/else and model.parameters() seems overkill I guess. Not sure if this could have been a problem in older PyTorch versions though.