The linear layers should be alright. You can check the gradients, if you have doubts, but the code looks fine.
I think since your loss is now bigger, the learning rate might be too high.
It’s speculative, but might be worth a try.