Here’s an idea…
To get more flexibility in your architecture you could use nn.ModuleList and construct the pipeline yourself.
Put all the layers in a list, and the shapes of the layers that follow in another list (or the shapes required as input by each layer):
  self.layers = nn.ModuleList(modules=[nn.Linear(,),  nn.Conv2d(,,,), ..])
  self.shapes = [(,,), (,,,),  ..]

then thread all the parts together:
def forward(self, x):
  for layer,shape in zip(self.layers,self.shapes):
    x = layer(x).reshape(shape) # reshape for the next layer if needed
  return x

I hope that helps.