@alexis-jacq
Aha, I see what you mean, yes each of the state S0/S1 should require gradient. What it was describing is that the parameters of the policy network should not require grads in S1, but need to require grads in S0. Because if we set all parameters requiring gradient, the parameters.grad will of a summation of twice. Think of 20 time step, when we backprop with c20.backward(), the policy network parameters will added up 20 times gradients.
It is not the case with TD learning, the summation of cost values at each time step is the objective function to backprop w.r.t the parameters of policy network. And the policy network produces continuous action directly.