It works well if you replace Variable(param.grad.data) with Variable(copy.deepcopy(param.grad.data)) in g and k
model.zero_grad() was zeroing out g and k because they share the same memory location as the gradients which get reset to 0.
copy.deepcopy will create a new location in memory that retains the values in g and k even after model.zero_grad() is called.
Here’s a link to working version:

  
      gist.github.com
  
  
    https://gist.github.com/ethancaballero/ab19ec0b3e5d8ab2a9f515b6125d6c80 33
trpo.py
# Adapted from https://github.com/pfnet/chainerrl/blob/master/chainerrl/agents/acer.py#L203
import torch
from torch import nn
from torch.autograd import Variable
from torch.nn import functional as F

def update_avg(shared_avg_model, model, alpha):
    for shared_avg_param, param in zip(shared_avg_model.parameters(), model.parameters()):
        shared_avg_param.data = shared_avg_param.data*alpha.expand_as(shared_avg_param) + (1-alpha).expand_as(param)*param.data

This file has been truncated. show original