Thanks for the replies so far.
Will there be a way in the future to call a custom autograd.function several times and each of these function call in the graph will know which saved_tensors belong to which call?
This would make it easier to write custom activation functions, as in the example given here: http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-defining-new-autograd-functions 11 and reuse them several times as it is possible with F.relu etc.
Of  course the function would just depend on the Input without internal parameters.