So if you want to optimize your policy (let call it P) with gradient descent, you should have variables taken as input (with is a state S) that requires a grad.
So, if in your code, you have something like this:
A = P(S1) # A is the action
S2, C = Dynamic(S1, A) # S' the new state, C the cost function you want to minimize
C.backward()
optimizer.step() # optimizer that updates P in order to optimize C

Then S1 should require a gradient, otherwise, the backward function applied to C will not know what to compute. If you don’t want to accumulate gradient (which I understand) just call optimizer.zero_grad() at the beginning of the gradient descent loop.