Solved by smth in post #2 
              
              
                you could look at something like https://github.com/facebookresearch/higher for this purpose. 
It functionalizes the model, where it’s parameters can be detached and backproped through