I was trying to do a moving average but was worried that it would negatively interfere with my backprop or something weird (sorry new to pytorch. If I do:
W = Variable(w_init, requires_grad=True)
W_avg = Variable(torch.FloatTensor(W).type(dtype), requires_grad=False)
for i in range(nb_iterations):
    #some GD stuff...
    W_avg = (1/nb_iter)*W + W_avg

would that be ok? Would it compute my average parameters and not interfere with other stuff?