@albanD I have another follow up question.
I was wondering if the forward activations are exposed in the nn.Module level?
I have seen activations need to be explicitly saved to the ctx if we are using the autograd.Function. But I am wondering how PT would keep the activations when someone’s using the nn.Module API?