So currently here are my code, knowing that i is for each sample
the part in learn method:
self.actor.optimizer.zero_grad()
action_q = self.actor.forward(states)
policy_output = -self.critic.forward(states, action_q)
        
for i in range(policy_output.size()[0]):
    policy_output[i].backward(retain_graph=True)
    for name, param in self.actor.named_parameters():
        self.params_grads[name] += invert_gradient(param.grad.data, action_q.detach()[i])
    ??? self.actor.optimizer.zero_grad() ???

for name, param in self.actor.named_parameters():
    param.grad.data = self.params_grads[name]/(self.batch_size*self.n_actions)
    self.params_grads[name] = T.zeros_like(param.data)

invert gradient
def invert_gradient(grads, action, lower_bound=-1, upper_bound=1):
    bound_range = upper_bound-lower_bound
    pdiff_max = T.sum(T.div(upper_bound-action, bound_range))
    pdiff_min = T.sum(T.div(action-lower_bound, bound_range))
    zeros_grad = T.zeros_like(grads)
    grad_inverter = T.where(T.gt(grads, zeros_grad), T.mul(grads,pdiff_max), T.mul(grads,pdiff_min))
    return grad_inverter

do you think this is correct?
so what is the zeroing gradient for each i will do, now knowing that i is for iterating sample size? will it zero gradients for each action?
or should I have another iteration inside here like this:
for i in range(policy_output.size()[0]):
    policy_output[i].backward(retain_graph=True)
    for a in action_q.detach()[i]:
        for name, param in self.actor.named_parameters():
            self.params_grads[name] += invert_gradient(param.grad.data, a)
        self.actor.optimizer.zero_grad()

without summing inside invert grad method.
thank you so much beforehand
EDIT: i think i found an example repo of this for you guys also who needed it, turns out it’s with all other paramterized RL agent in pytorch too : GitHub - cycraig/MP-DQN: Source code for the dissertation: "Multi-Pass Deep Q-Networks for Reinforcement Learning with Parameterised Action Spaces" 6
I’m going to read it and try it out