Thanks. Your suggestion looks clean and I like the idea to just have one function in the class. I will just return a dictionary with all the outputs in the forward function:
def forward(x):
    outs = {}
    fc1 = self.fc1(x)
    outs['fc1'] = fc1
    relu1 = F.relu(fc1)
    outs['relu_fc1'] = relu1
    ...
    return outs