Suppose we have our computational graph like the figure above (second c1 should be c2), should we actually set dynamics.parameters.requires_grad=False and policy.parameters.requires_grad=False ?
Since dJ/dpolicy = dc1\dpolicy + dc2\dpolicy = dc1\ds1ds1\ddynamicsddynamics/da0da0/dpolicy + dc2/ds2ds2/ddynamics*[ddynamics/ds1 + ddynamics/da1]
= dc1\ds1ds1\ddynamicsddynamics/da0da0/dpolicy + dc2/ds2ds2/ddynamicsddynamics/ds1ds1/da0da0/dpolicy + dc2/ds2ds2/ddynamicsddynamics/da1da1/dpolicydpolicy/ds1ds1\ddynamicsddynamics/da0da0/dpolicy
So I guess I just the upward gradients from each cost along through the first action a0,  According to the formula above, the gradients of policy shouldn’t be accumulated until it reaches the first action, and gradients of dynamics model should not accumulated at all, just let the local-gradients flow from upward to downward (i.e. without += dynamics.parameters.grad)