Fine-tune distilbert-base-uncased under run_glue
## ❓ Questions & Help

I am a bit confused on how to fine-tune distilbert. I see three options

1. Fine-tune bert for the task and then use distillation.distiller 
2. Fine-tune bert for the task and then use distillation.train 
3. Fine-tune distilbert-base-uncased directly for the task using run_glue.py

I tried the 3rd option under run_glue.py as follows

add distilbert to MODEL_CLASSES 
```MODEL_CLASSES = {
    'bert': (BertConfig, BertForSequenceClassification, BertTokenizer),
    'distilbert': (DistilBertConfig, DistilBertForSequenceClassification, DistilBertTokenizer),
    'xlnet': (XLNetConfig, XLNetForSequenceClassification, XLNetTokenizer),
    'xlm': (XLMConfig, XLMForSequenceClassification, XLMTokenizer),
    'roberta': (RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer),
}
```
and add a flag `--model_type=distilbert`

Which of the three methods above should be used?
Hello @YosiMass,
The simplest/more direct way to do transfer learning is indeed the 3rd solution.
If you use `run_glue.py`, the modification you made is correct. You also have to be careful since DistilBERT doesn't take `token_type_embeddings` as input --> [here](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py#L131) and [here](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py#L221).
I'll add these modifications in a few days directly to these scripts so that it's seamless to use DistilBERT with run_squad or run_glue.<|||||>Thanks @VictorSanh.  

Yes, I handled the missing token_types as follows. In run_glue.train and run_glue.evaluate I changed From  
```            
inputs = {'input_ids':      batch[0],
                'attention_mask': batch[1],
                'token_type_ids': batch[2] if args.model_type in ['bert', 'xlnet'] else None,  # XLM and RoBERTa don't use segment_ids
                'labels':         batch[3]}
```

To
```
if args.model_type not in ['distilbert']:
     inputs = {'input_ids':      batch[0],
                     'attention_mask': batch[1],
                     'token_type_ids': batch[2] if args.model_type in ['bert', 'xlnet'] else None,  # XLM and RoBERTa don't use segment_ids
                     'labels':         batch[3]}
else:
      inputs = {'input_ids':      batch[0],
                      'attention_mask': batch[1],
                      'labels':         batch[3]}
```<|||||>> handled the missing token_types as follows. In run_glue.train and run_glue.evaluate I changed From

It looks good to me!