If I understand the use case correctly, you wouldn’t need to flatten the activation of the last layer, as it already is 2-dimensional (has the shape [100, 20]).
You could just add two more linear layers afterwards, where the first uses in_features=20.