Hi @nielsr

This will be updated in a PR I’m currently working on, such that the decoder_input_ids will be created automatically based on the labels provided by the user.

Is this the current behavior for BART? I find it weird that the output loss is the same regardless of providing or not the decoder_input_ids, unless these are being generated internally in the model from the provided labels. However, if that’s the case, this behavior seems undocumented. The docs say that:
If no decoder_input_ids is provided, the model will create this tensor by shifting the input_ids to the right for denoising pre-training following the paper. (BART — transformers 4.11.3 documentation)
Minimal code to reproduce the issue:
from transformers import AutoConfig, AutoModelForSeq2SeqLM
import torch
from torch import tensor

config = AutoConfig.from_pretrained(
            'facebook/bart-large',
            cache_dir=None,
            revision='main',
            use_auth_token=None,
        )
bart = AutoModelForSeq2SeqLM.from_pretrained(
            'facebook/bart-large',
            from_tf=False,
            config=config,
            cache_dir=None,
            revision='main',
            use_auth_token=None,
       )
          
input_ids = tensor([[0, 510, 1290, 1043, 3019, 6, 1261, 36, 16256, 43, 3]])
attention_mask = torch.ones_like(input_ids)
labels = tensor([[0, 24476,  6302, 16629, 5,  5794, 160, 17223, 1043,  3019, 2]])
decoder_input_ids = tensor([[2, 0, 24476, 6302, 16629, 5, 5794, 160, 17223, 1043,  3019]])

bart.eval()  # just to make it deterministic

# loss when decoder_input_ids are provided
out1 = bart(input_ids=input_ids, attention_mask=attention_mask, decoder_input_ids=decoder_input_ids, labels=labels)
loss1 = out1.loss

# loss when decoder_input_ids are not provided
out2 = bart(input_ids=input_ids, attention_mask=attention_mask, decoder_input_ids=None, labels=labels)
loss2 = out2.loss

print('Loss with decoder_input_ids   :', loss1.item())
print('Loss without decoder_input_ids:', loss2.item())