@mathematicsofpaul  I don’t think the problem was with the position encoding. I think it was the initialisation. I’d copied the code below and applied it to the full model - it appears to mess something up  - when I removed it and added the position encoding back training converges to a much lower loss and the decoder stops repeating the same token over and over.
for p in model.parameters():
    if p.dim() > 1:
        torch.nn.init.xavier_normal_(p)

Perhaps the code overwrites the position encoding as it’s stored in a buffer - although the docstring for ‘register_buffer’ does say it’s for model state that aren’t parameters? The other option is the embedding weights are initialised to zero - perhaps overwriting them with non-zeros messes with that layer?