I am trying to implement cell gradient clamp for nn.LSTM.
Specifically, I need to clamp cell.grad whenever it is computed for every time step.
This is implemented in Alex grave’s rnnlib implementation.
(https://sourceforge.net/projects/rnnl/files/ 14 > LstmLayer.hpp line 315)
//constrain errors to be in [-1,1] for stability
if (!runningGradTest)
{
	bound_range(inErrs, -1.0, 1.0);
}

This is different from calling torch.nn.utils.clip_grad_norm() after loss.backward().
Inspired by Gradient clipping 22
and https://github.com/DingKe/pytorch_workplace/blob/master/rnn/modules.py#L122 8,
I was trying to set register_hook() on ‘hidden’ variable to modify gradient on-the-fly. https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py 8 : line 181
Here is the example code :
def clamp_grad(v, min, max):
    v.register_hook(lambda grad: grad.clamp(min, max))
    return v

output, hidden = func(input, self.all_weights, hx, batch_sizes)
hidden = clamp_grad(hidden, -1.0, 1.0)

The problem I’m having now is ‘hidden’ variable has Nonetype grad at backward phase.
(i.e. v.grad == None)
Is there any other way to implement cell gradient clamp?