@calebh I believe that there is something going on with the tgt_mask mechanisms with the generate_square_subsequent_mask or something related to that matter. Clearly, if we are getting repeated tokens then that means that it is taking the whole sentence of words/embeddings as opposed to taking them sequentially with the masking.