Everything is exactly the same now and it has made it worse.
batch_size = 32
learning_rate = 1e-4
decay =  1e-6
seed = 2
nn.CrossEntropyLoss()
optim.Adam(net.parameters(), lr=learning_rate, weight_decay=decay)
Train Loss: 4.848 | Accuracy: 5.459
Train Loss: 4.806 | Accuracy: 31.017
Train Loss: 4.659 | Accuracy: 29.777
Train Loss: 4.401 | Accuracy: 70.471
Train Loss: 4.188 | Accuracy: 70.471
Train Loss: 4.149 | Accuracy: 70.471
Train Loss: 4.159 | Accuracy: 70.471
Train Loss: 4.164 | Accuracy: 70.471
Train Loss: 4.164 | Accuracy: 70.471