Gradients for model parameters could be accessed directly (e.g. self.conv1.weight.grad). What about gradients for activations?
I use ReLU activations, so I technically I could use gradients for biases. The problem is I don’t use biases in my network. So, short of computing the entire chain of gradients manually, is there a way to get them from autograd?
Why do I need them? I want to use a learnable threshold for ReLU clipping:

relu1 = torch.where(relu1 > thr, thr, relu1)

where thr is a trainable model parameter. The threshold function is not differentiable, so I want to estimate its gradient from the gradients of the activations. The gradient for thr should be proportional to the sum of gradients for all activations.
Any ideas of how to do it better are welcome!