Hey David. Thanks for looking into it. I have been trying a few things too. So I applied a softmax activation to the outputs. And still no luck. The loss does decrease but again starts increasing even before the first epoch is complete. I tried stopping the model at the lowest loss and had it generate text from a seed in the training, and it spits out complete gibberish.