Sorry I should have been more explicit.
You have to use ModuleList — PyTorch 1.8.1 documentation 2 which is a list-like module which is properly tracked by the engine. Think that attribute of a nn.Module distinct from a tensor or another module is not tracked by the engine.