The code snippet looks alright (although you don’t need to wrap encoder_decoder in another class, I assume your code is more complicated).
All submodules in encoder_decoder should be scattered to the GPUs during the forward pass.
Did you push any parameter or tensor inside your custom modules to a specific GPU?
This would break the automatic scattering, as you would be forcing this tensor to be e.g. on GPU0.