This is expected behavior, if you see BartforConditionalGeneration docs, the decoder_start_token_id is the EOS token by default (which is ). This is in contrast to other models, a BERT decoder for instance has a CLS token as the starting token given to the decoder.