Solved by ptrblck in post #4 
              
              
                No, ReLU does not contain any parameters. They correspond to the nn.Batchnorm layer in the encoder. 
The index in encoder.model.1 is corresponding to the ModuleList index when you print the model.