Sorry i misinterpreted the question. So as i understand you are trying to make the Consensus Optimizer, but in pytorch. The code would be something like this:
class ConsensusOptimizer(object):
    def __init__(self, learning_rate, alpha=0.1, beta=0.9, eps=1e-8):
        self.optimizer = torch.optim.RMSPropOptimizer(learning_rate)
        self._eps = eps
        self._alpha = alpha
        self._beta = beta

    def conciliate(self, d_loss, g_loss, d_vars, g_vars, global_step=None):
        alpha = self._alpha
        beta = self._beta

        # Compute gradients
        d_grads = torch.autograd.grad(d_loss, d_vars)
        g_grads = torch.autograd.grad(g_loss, g_vars)

        # Merge variable and gradient lists
        variables = d_vars + g_vars
        grads = d_grads + g_grads

        # Reguliarizer
        

        reg = 0.5 * sum(
            torch.sum(torch.square(g)) for g in grads
        )

        # Jacobian times gradiant
        Jgrads = torch.autograd.grad(reg, variables)
       
        for param in variables:
               param.grad+=self._alpha*Jg
        optimizer.step() 

I assume you call this after you call loss.backward(). This is untested code and you prob need to make some changes.
See more info here 2