Based on the error message some parameters of self.value_net didn’t receive a gradient and thus their .grad attribute is set to None.
This could happen, e.g. if you’ve (accidentally) detached the computation graph or if these parameters were not used in the forward pass (and would thus not receive a gradient).