DeBERTa pretraining data preparation
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: 4.6.0.dev0
- Python version: 3.6
- PyTorch version (GPU?): 1.6
- Tensorflow version (GPU?):
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: Y

### Who can help
 --> @LysandreJik @BigBird01 




## Information

Model I am using (Bert, XLNet ...): DeBERTa

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name): MLM + SQUAD 1
* [ ] my own task or dataset: (give details below)



<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I am pretraining DeBERTa Base from scratch on Wikipedia + Book Corpus dataset. After pretraining for 500K steps I observe SQUAD 1.1 score of 76 which is much less than Figure 1(b) in paper (although figure 1b reports squad 2.0 numbers) squad 1.1 numbers would be much better than that as it is easier task. Using same hyperparameters as reported in paper. I would like to confirm the preprocessing steps that authors took to prepare pretraining data. 

1. In section 4.4.1, authors report that they used Megatron code base to deduplicate the data.  The code provided performs deduplication based on urls. https://github.com/NVIDIA/Megatron-LM/tree/main/tools/openwebtext Was the deduplication performed on url -> document set or on shards of dataset? 
2. [This](https://github.com/NVIDIA/Megatron-LM/blob/main/tools/openwebtext/cleanup_dataset.py) codebase also cleans up the dataset based and removes non-english characters. Were these data cleanup steps performed on pretraining data? 
3. Is it possible to provide scripts used to generate pretraining data? 
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.