I solved my problem by:

Not making activs and outputs nn.Parameters
Not assigning them as model attributes. Instead I added them as optional key word arguments in the forward method and returned the activations as well.