Train GPT2 with Trainer & TrainingArguments using/specifying attention_mask
Hi, I'm using Trainer & TrainingArguments to train GPT2 Model, but it seems that this does not work well.

My datasets have the ids of the tokens of my corpus and the mask of each text, to indicate where to apply the attention:
```
Dataset({
     features: ['attention_mask', 'input_ids', 'labels'],
     num_rows: 2012860
 }))
```
I am doing the training with Trainer & TrainingArguments, passing my model and my previous dataset as follows. But nowhere do I specify anything about the attention_mask:
```
training_args = TrainingArguments(
    output_dir=path_save_checkpoints,
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size = 4, 
    gradient_accumulation_steps = 4, 
    logging_steps = 5_000, save_steps=5_000,
    fp16=True,
    deepspeed="ds_config.json",
    remove_unused_columns = True,   
    debug = True
    )


trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    tokenizer=tokenizer,
)

trainer.train() 
```

How should I tell the Trainer to use this feature (attention_mask)?
If you take a look at the file /transformers/trainer.py there is no reference to "attention" or "mask".

Thanks in advance!
Hi there,

If your dataset or collator returns the `attention_mask`, then you don't need to pass it separately. With `Trainer`, all the input that you want to pass to model's `forward` should be returned by the dataset/collator and it will be passed to `model.forward` by `Trainer`.<|||||>Thank you very much for your quick reply @patil-suraj 

Is it possible that the problem is that more than 50% of the input is padding? Could this be too much? Do you think that training more would solve it? It currently returns a minimal loss. Having such a low loss is what made me think that I was not considering the `attention_mask` and that is why the loss was low (obviously it is easy to predict 600 padding tokens xD)

What I am doing now is changing the size of the input, instead of 1024 (default value) I am testing with 400 (size of the longest text in my dataset).

Best regards!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.