How to preprocess my own data to use RoBERTa of Multiple GPUs
Hey,

I am bit naive using deep learning of text-classification, my data **(.csv)** consist of basically two columns:

- Text
- Labels

As per basic objective, model should take unseen text and predict label _(variable y)_ from the trained labels.

**I followed this tutorial to train RoBERTa algorithm:**
- [https://colab.research.google.com/drive/1xg4UMQmXjDik3v9w-dAsk4kq7dXX_0Fm](https://colab.research.google.com/drive/1xg4UMQmXjDik3v9w-dAsk4kq7dXX_0Fm)

Here the input format is universal (Train.tsv and Test.tsv) with 2 columns (which I mentioned above). The only problem is that this code doesn't utilize the multiple GPUs **(I even tried the DataParallel wrapper)**. 
Somehow, I found the repository of the pytorch-transformer where they have given an example of how to utilize multiple GPUs and train the RoBERTa model i.e:

- [https://github.com/huggingface/pytorch-transformers/tree/master/examples](https://github.com/huggingface/pytorch-transformers/tree/master/examples)
This repository example takes the input-data in the wiki.text data format and they have provided the link [ Pretraining RoBERTa using your own data ](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md#pretraining-roberta-using-your-own-data) and **yet I don't find it of any use as it just talks about the format what they think is right not about the transforming of the standard format (i.e: .CSV)**

Here is the sample:
```
 = Valkyria Chronicles III = 
 
 Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to as Valkyria Chronicles III outside Japan , is a tactical role @-@ playing video game developed by Sega and Media.Vision for the PlayStation Portable . Released in January 2011 in Japan , it is the third game in the Valkyria series . <unk> the same fusion of tactical and real @-@ time gameplay as its predecessors , the story runs parallel to the first game and follows the " Nameless " , a penal military unit serving the nation of Gallia during the Second Europan War who perform secret black operations and are pitted against the Imperial unit " <unk> Raven " . 

```

I mean, why? do they have to use this format, why can't they go with standard format of classification algorithm? and if they want to then why they don't provide the detail regarding the data and adjusting the personal data corresponding to the algorithm intake?

Just wanted to know, how to preprocess the data according to multiple GPUs RoBERTa example, any help would be appreciated.

thanks.

@spolu @cynthia @myleott <|||||>Hi, you can follow the `run_glue` example which is better for text classification.
But you will have to modify it for your needs, it's not plug and play.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
