@mathematicsofpaul - one thing I just noticed after banging my head against this for the last day - I removed the PositionalEncoding layer (from https://pytorch.org/tutorials/beginner/transformer_tutorial.html 31) and that “fixed” the problem. Before I did this it simply repeated the same token (both training and validation).
Not sure if there’s a bug in it, I suspect that it’s necesary for a real transformer (I’m just trying to get it to work with a couple of training cases and using the same cases for validation) so I guess I’ll have to look closer at what it’s doing (and what it’s supposed to do)