Try to overfit a small dataset (e.g. just 10 samples) by playing around with the model architecture as well as the hyperparameters first. Your current model contains a lot of stacked linear layers and I don’t know if you’ve already verified that the model is not suffering from e.g. vanishing gradients etc.