Solved by albanD in post #8 
              
              
                If your eval network is just a net with requires_grad=False. It won’t present the output from requiring gradients if the input does. So if the input to the eval net requires grad, then the output will as well. 
And if there is a function that computes these values based on the action, then they shou…