I had the impression, knowing the dynamic nature of graphs in pytorch, adding a variable to the graph would automatically include it in the parameter list and thus in the optimization process!
It seems my understanding is flawed here.