I mean that you can write a custom nn.Module for both, encoder and decoder. You can return several outputs in decoder class and to have several inputs in forward. This way you can keep the packaged format you have.