Before a professional person comes here to explain.
I think you misuse the nn.linear:
You can read this example or read this How to use nn.linear 18
    def __init__(self):
        super(Network, self).__init__()
        self.linear1 = nn.Linear(32, 64)  #32 Input-Neurons, 64 Output-Neurons, Linearer Layer
        self.linear2 = nn.Linear(64, 128) #64 Input-Neurons,128 Output-Neurons, Linearer Layer

I just want to share what I learned. If you are an increasing amount of layers and neurons, you are just an increasing amount of variable or feature. But the problem is your input data does not have that much feature for you to extract. I am not really sure. I remember it is also called the overfitting problem.
In addition, even though your input data has that many features to be extracted, you may have a gradient explosion and vanishing problem.
If that so, we need something to can learn to turn your model smaller or bigger when it needs to:
It is called ResBlock
Then try to put those ResBlocks between every layer
Reference:
class ResBlock(nn.Module):
    def __init__(self, nf):
        super().__init__()
        self.linear1 = conv_layer(nf,nf)
        self.linear2 = conv_layer(nf,nf)


Concept of ResBlock and how to use it to improve it by adding more layers:
Lesson 7: Deep Learning 2019 - Resnets from scratch; U-net; Generative...
Example code for using ResBlock: https://colab.research.google.com/drive/1NVxx0E6GoX9vzaigCs1urVN3529DsBtn#scrollTo=n3r8ZHuObvXu 2
If you want, please share it in Colab notebook or the data set to me and I will try to use a basic linear layer to train or possible ResBlock to improve it if I have time.
I feel you don’t really understand the nn.linear layer. Try to read this: https://stackoverflow.com/questions/54916135/what-is-the-class-definition-of-nn-linear-in-pytorch 18