@alexis-jacq
c1 and c2 are cost values of a state. This is actually a direct policy search method. The goal is to optimize the policy which can reduce the summation of costs at each time step. It seems to me that the dynamics model shouldn’t accumulate gradient with (requires_grad = False) just not to keep adding up dynamics.parameters.grad which might either explodes or provide the wrong gradient to policy network ?
However, in this way, the natural solution should be even the policy network should not accumulate gradient for each time step, only store the gradients to policy.parameters.grad when it reaches the first action a0. This makes me a bit confused since we can only either set requires_grad = False/True to policy network in one time, I don’t know how to set False for all other time step and True only for first step in a0 , where the same network is used.