You definitely should tokenize your labels, but remove the lines
samples["decoder_input_ids"] = labels.input_ids
samples["decoder_attention_mask"] = labels.attention_mask

which the data collator will ser properly from the labels and the models (there is a shift right to apply in some models).