Hi everyone,
I know it’s been too long since the last update but I just had the problem with inverting gradient so I wanted to ease this subject for anyone else.
as @gemsanyou said the inverting gradient is implemented in mp-dqn 3. For anyone who encounters this problem in future i will explain the code here.
this being the inverting gradient function:
	def invert_gradient(self,delta_a,current_a):
		index = delta_a>0
		delta_a[index] *=  (index.float() * (self.max_p - current_a)/self.rng)[index]
		delta_a[~index] *= ((~index).float() * (current_a- self.min_p)/self.rng)[~index]

		return delta_a	

gets the gradient(wrt actions) and actions themselves. index is a masking variable indicating gradient is suggesting increment of action or not. gradients are then updated accordingly.
Now in order to use this function:
current_a = Variable(self.actor(state))
current_a.requires_grad = True
actor_loss = self.critic(state, current_a).mean()
self.critic.zero_grad()
actor_loss.backward()
delta_a = deepcopy(current_a.grad.data)
delta_a = self.invert_gradient(delta_a,current_a)
current_a = self.actor(state)
out = -torch.mul(delta_a,current_a)
self.actor.zero_grad()
out.backward(torch.ones(out.shape).to(device))

First you get the gradient wrt actions then after applying inverting gradient you will end up with modified dQ/da now what did we want in the first place? -(dQ/da) * (da/dw)
we changed dQ/da part now simply consider dQ/da = c. having -ca calculating gradient wrt weights we’ll end up with -c(da/dw) = -(dQ/da) * (da/dw).
One point worth mentioning here is about out.backward and why it has an ones vector as its input. it is because “a” is a vector and in backward we will end up with Jacobian matrix however for updating the weights we need one vector of gradients so we simply give the linear combination of J matrix’s columns as the input(one could pass delta_a to backward however it is not implemented like this in the original code) for more info read gradient arg 6