Nice explanation! @Florian_1990



 Florian_1990:

One thing you could do, is to check if the input ( a_ ) requires gradients and use torch.no_grad() instead of torch.enable_grad() or skip the requires_grad_() part to prevent the calculation of some unnecessary gradients.


It might be easiest to have a wrapper (you wouln’t use MyFn.apply as a name, anyway) that checks whether torch.is_grad_enables() and whether the inputs need the gradient. And pass that as a boolean argument to the function. At least that’s what I usually do for extensions.
Best regards
Thomas