Using the autograd.Function instance only once is a great way to do this. In fact this is what used to happen all the time when you used operations on variables and is the right thing to do™.


Ok, probably it is a way to go, where do I find an example with instantiating function?


If you absolutely dislike autograd.Function instances or - like myself - like to feel modern, go for the new-style autograd.Function and change the line with the function application to y_pred = MyReLU.apply(x.mm(w1)).mm(w2). You do not need relu when doing this. This will store inputs and other stuff in the context ctx and is what happens nowadays when you use operations on variables.


Values that I need to store are actually integer numbers that occur during forward pass, so I can’t follow this recipe.