Train RobertaModel from scratch for my dataset
I am trying to train RobertaModel from scratch. I am following [this](https://huggingface.co/blog/how-to-train) blog but instead of `model = RobertaForMaskedLM(config=config)`, I am starting with `configuration = RobertaConfig()
model = RobertaModel(configuration)` and then continuing with other steps. But I am getting error `TypeError: forward() got an unexpected keyword argument 'labels'`. The whole code piece:
```
configuration = RobertaConfig()
model = RobertaModel(configuration)
from transformers import LineByLineTextDataset
dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="./train.txt",
    block_size=128,
)
from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=False, mlm_probability=0.15
)
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./Model1",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_gpu_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    prediction_loss_only=True,
)
trainer.train()
```
Is there some other way to do pre-training? Am i missing something here?
Are you using transformers from source or from pip install ?
`labels` is introduced in a recent commit and available on master, if you  are on version 2.11.0 from pip install then use `lm_labels`<|||||>> Are you using transformers from source or from pip install ?
> `labels` is introduced in a recent commit and available on master, if you are on version 2.11.0 from pip install then use `lm_labels`

@patil-suraj  yes I am on version 2.11.0 so where should I use `lm_labels`<|||||>If you installed using pip then yes, use `lm_labels`

You'll need to change `DataCollatorForLanguageModeling`, also as you are using `Roberta` which means you training it for maksed language modelling, so you'll need set `mlm `to `True` <|||||>but I am training a RobertaModel not a RobertaMaskedLM model.<|||||>What is your pre-training objective ? Roberta is pre-trained using masked language modelling objective <|||||>i am training it from scratch for my own dataset. I want to use the vectors obtained from last layer for classification task<|||||>You can train `RobertaForMaskedLM` using the `mlm` objective and then load it in `RoberatForSequenceClassification` for classification .
`RoberatForSequenceClassification` will take care of taking last layer vector and feeding it to a classification layer.<|||||>yes I have done that now I want to compare it with others RF or GDBTs and extract some features and that's why I want to train RobertModel.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
