Yes,state_action_values are indeed the output of the policy_net, which has the input state_batch. I checked and state_action_values.requires_grad=True even though it was never explicitly written in code. (I guess this is by default when passing tensors through models, unless with torch.no_grad() - right?)