What you typically do with an imbalanced set in a classification problem is using class weights in your loss function. See the documentation of CrossEntropyLoss 1 and its weight parameter. However, I do not think that the Trainer currently allows custom loss functions out-of-the-box. Instead you can subclass the Trainer, particularly overwriting the compute_loss method to calculate the loss manually.

  

      github.com
  

  
    huggingface/transformers/blob/c02cd95c56249e9bd38ecb3e4ebcce6d9eebd4a4/src/transformers/trainer.py#L1811 1



      
      
                  with amp.scale_loss(loss, self.optimizer) as scaled_loss:
                      scaled_loss.backward()
              elif self.deepspeed:
                  # loss gets scaled under gradient_accumulation_steps in deepspeed
                  loss = self.deepspeed.backward(loss)
              else:
                  loss.backward()
          
          
    return loss.detach()
          
          
def compute_loss(self, model, inputs, return_outputs=False):
              """
              How the loss is computed by Trainer. By default, all models return the loss in the first element.
          
          
    Subclass and override for custom behavior.
              """
              if self.label_smoother is not None and "labels" in inputs:
                  labels = inputs.pop("labels")
              else:
                  labels = None
              outputs = model(**inputs)