Thanks so much for your kind words and suggestions.
I’ve made a lot of progress and my model is now learning. I’m not 100% clear what made the difference, but your recommendations were very helpful in sorting things out.

Check that the embedding parameters are getting gradients

They were indeed getting gradients.

Check the loss. Your model output is linear, so the loss should apply a softmax (i.e. use cross entropy loss).

I’m using nn.CrossEntropyLoss, which includes a softmax.

Disable the position encoding

What I did instead was to swap my position encoding implementation into former, and it didn’t hurt its learning.

Try it with 0 transformer layers (i.e. just train word embeddings).

This was a great recommendation and was really informative. Both former and my model learned well with 0 transformer layers. Then I found that my model would learn okay with 1 or 2 layers, but at 3 was having trouble. Eventually I found that decreasing my learning rate from 0.001 to 0.0001 and extending my training time from 10 to 50 epochs showed that it was eventually learning. With some tinkering with the gradient clipping, the LambdaLR scheduler, and the initial learning rate, I now have its accuracy on the training set eventually reaching high 90s. I know I had taken learning rate down to 1e-5 previously to ensure that wasn’t the issue; my guess is that there were multiple problems and each one masked seeing the improvement when I tinkered with the other.
Now that I can see that my model is capable of learning I consider this issue solved. Thank you so much!
I will update soon to post my actual model in case it’s helpful for future readers.