Are you framing your classification problem as a sequence generation task? what types of labels do you have for your training data? Are the labels text/sequence or a finite number of categories? If your task is classification I believe you’re using the wrong model class. You could probably use BertForSequenceClassification for a sentiment analysis task as has been done in the link below:

  

      huggingface.co
  

  
    

nlptown/bert-base-multilingual-uncased-sentiment at main 3



  

  
    
    
  

  


And instead of using Seq2SeqTrainer, just use Trainer and TrainingArguments.