I found my problem. For anyone else after me, I had a loop making multiple passes through the model and adding each pass’s loss to a running total. But I initialized the total loss to 0 outside the loop and that tensor had requires_grad=False, a property that was preserved as each pass’s loss was added to the total loss. I solved this using total_reward = torch.zeros(1, requires_grad=True).