Solved by ptrblck in post #2 
              
              
                Generally you could use nn.Modules, which will keep the parameters as internal attributes, or use the functional API where you can keep track of all parameters manually. 
I would recommend to take a look at some tutorials to see different work flows and use cases.