Actually, that was causing a memory leak.  I switched to a register_hook instead which also passes my value checker but does not work on 2 GPUs.  Code for that is as follows:
class valueTracker(nn.Module):
    def __init__(self, out_channels):
        super(valueTracker, self).__init__()
        self.register_buffer('average', torch.zeros(out_channels, device=gf.device, dtype=torch.double))

Then in my module
    #in init
        self.values = nn.ModuleList([])
        self.values.append(valueTracker(self.out_channels))
    ...
    def forward(self,x):
        ...
        out.register_hook(lambda grad: saveAverageD(grad, self.values, anotherTensor))
        return out

and the original module backward hook that is now a tensor hook:
def saveAverageD(grad_out, Values, additonalTensor):
    with torch.no_grad():
        Values[0].average = Values[0].average * 0.99 + grad_out.sum((0,2,3)) * 0.01