Solved by tom in post #2 
              
              
                These will not show up in your list(model.names_parameters()) and thus not be trained. Use a torch.nn.ModuleList to fix that, or, at your style preference, add the ReLUs there and make it a torch.nn.Sequential model… 
Best regards 
Thomas