I think my issue then lies with preprocessing. I thought both input and output sequences had to start with SOS and end with EOS, but from what you are saying it seems that the input should contain SOS and EOS but output should only contain EOS?
EDIT: After going through the tutorial again, I found that they indeed only add EOS tokens in preprocessing and SOS is only added in the decoder. I always thought, having experience from language models, that SOS would also be added in the input.