I don’t know how and when each code snippet is called, but check the .grad attribute of policy.action_head before and after the first backward call and make sure its set to None first and shows a valid gradient afterwards (remove the zero_grad call if it was used before).
If you are seeing valid gradient values (even if they are close to zero) the gradients are indeed calculated but might be too small to see a significant change and you should then debug the used operations.