The Module is just a submodule of my whole network. I joined the parameters with self.y using intertools.join when giving the parameters to the optimizer. This is ugly but it was just to test a idea really fast. So I am still confused why my original way did not work.