Thanks for your help, but my question is a little different
In my formula A* is a set of all possible episodes
when I update parameters , I want something like
policy * grad of loss
or in other words
(p1loss(p1) + …+pNloss(pN)) , where p1+…+pN = 1
your solution is
(loss(p1)+…+loss(pN)) / N
I want to change grad between loss.backward() and  optim_policy.step()
I tried loss.grad() but it is None