So I finally got it working. I had to remove the activation on the output layer and the loss decreases like expected. But, it was just a quick fix. Doesn’t make any sense. Assuming a multinomial distribution of characters, there has to be a softmax at the end to output a distribution to sample from. And I had to remove the softmax to get it to work. This is just bizarre.