Emm, I’m developing a strategy of moderating the gradient which needs the mean and variance of gradients of Convolution layer in a period of time. The series of gradients of weights and bias of conv2d in the training process are what I want to record. But I find it high memory-cost and replace it with the exponentially weighted moving average method in the last, just like what is used in batchnorm layer.
Though, changing the gradient in a postprocessing step in training loop is very clear,  it’s hard to locate the layer I want to change its gradient.
However, your advice offer me a new way of affecting the backward process and makes me understand the details better. Thanks!