If your eval network is just a net with requires_grad=False. It won’t present the output from requiring gradients if the input does. So if the input to the eval net requires grad, then the output will as well.
And if there is a function that computes these values based on the action, then they should require grad. Unless you detach/unpack Variable which you shouldn’t do.