Solved by justusschock in post #2 
              
              
                The point is not that they have to be registered during __init__ but before you create the optimizer since references to the parameters are given to. The optimizer. 
For example you could have a function build_model which creates the layers etc. and as long as you call it before creating the optimiz…