Hello everyone!
I tried to apply the technique of data augmentation, mixup, popularly used on computer vision, but in this case on NLP.
The algorithm developed is in two phases:
The first phase gets the representation for each sentence of the batch, computing the mean of the correspondent hidden states of the last layer. The fragment below shows the corresponding module.
class LanguageModel(nn.Module):
 
  def __init__(self, pretrained_model_name, device="cuda:0", anonymized_tokens=False):
    super(LanguageModel, self).__init__()
    # Load tokenizer
    self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name)
    # Load model
    self.config = AutoConfig.from_pretrained(pretrained_model_name)
    self.config.output_hidden_states = True
    self.model = AutoModel.from_pretrained(pretrained_model_name, config=self.config).to(device)

  def forward(self, input_ids, attention_mask):
    outputs = self.model(
        input_ids=input_ids,
        attention_mask=attention_mask,
    )
    activations = torch.mean(outputs[0], axis=1)
    return activations

After that, it generates the mixup examples using the function proposed on the original code 3, but  being  the input, the representations computed on the previous step, instead the images, like originally.
One time the mixup examples are generated, the second phase realizes the predictions (the fragment below shows the corresponding module). Finally, is computed the loss, in the same way as in the original work.
class ClassifierLayer(nn.Module):
 
  def __init__(self, num_classes, dropout_rate=0.1, petrained_size=768, device="cuda:0"):
    super(ClassifierLayer, self).__init__()
    self.layer = nn.Linear(petrained_size, num_classes, bias=True).to(device)
    self.drop = nn.Dropout(dropout_rate)

  def forward(self, z):
    activations = self.layer(self.drop(z))
    return activations

In the fragment of the code below, is shown a summary of the training loop proposed, however the full script used is here 3:
    for idx_epoch in range(0, args.num_train_epochs):
        language_model.train()
        classifier_layer.train()
        accs = 0; ps = 0; rs = 0; f1s = 0; lss = 0
        for (idx_batch, train_batch) in enumerate(train_dataloader):
            # 0: input_ids, 1: attention_mask, 2:token_type_ids, 3: labels
            batch_train = tuple(data_.to(device) for data_ in train_batch)
            labels_train = batch_train[-1]
            inputs = {
                'input_ids': batch_train[0],
                'attention_mask': batch_train[1],
            }
            optimizer.zero_grad()
            # 1st phase: conextual embeddings
            contextual_embeddings = language_model(
                input_ids=inputs['input_ids'],
                attention_mask=inputs['attention_mask'],
            )
            # 2nd phase: mixup
            inputs, targets_a, targets_b, lam = mixup_data(contextual_embeddings, labels_train, args.alpha_mixup, use_cuda)
            inputs, targets_a, targets_b = map(Variable, (inputs, targets_a, targets_b))
            predictions = classifier_layer(inputs)
            loss = mixup_criterion(criterion, predictions, targets_a, targets_b, lam)
            
            # 2nd phase: standard
            # predictions = classifier_layer(contextual_embeddings)
            # loss = criterion(predictions, labels_train)
            
            lss += loss
            loss.backward()
            optimizer.step()
            scheduler.step()

Experimenting with this approach, the results obtained are very poor…
Have any of you worked on an approximation similar to this one with successful/good results?
Thanks.