The aspect you would like to simplify is mostly the number of in_features in nn.Linear layers as far as I understand.
One way would be to use nn.AdaptiveMaxPool2d 2 or just convolutions.
Of course you could try to calculate the shape of in_features using the shape formula for conv layers etc.