readme: ---
  language: zh
  datasets: CLUECorpusSmall
  widget: 
  - text: "北京是[MASK]国的首都。"
  
  
  ---
  
  
  # Chinese RoBERTa Miniatures
  
  ## Model description
  
  This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).
  
  [Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users to reproduce the results, we used the publicly available corpus and provided all training details.
  
  You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:
  
  |          |           H=128           |           H=256           |            H=512            |            H=768            |
  | -------- | :-----------------------: | :-----------------------: | :-------------------------: | :-------------------------: |
  | **L=2**  | [**2/128 (Tiny)**][2_128] |      [2/256][2_256]       |       [2/512][2_512]        |       [2/768][2_768]        |
  | **L=4**  |      [4/128][4_128]       | [**4/256 (Mini)**][4_256] | [**4/512 (Small)**][4_512]  |       [4/768][4_768]        |
  | **L=6**  |      [6/128][6_128]       |      [6/256][6_256]       |       [6/512][6_512]        |       [6/768][6_768]        |
  | **L=8**  |      [8/128][8_128]       |      [8/256][8_256]       | [**8/512 (Medium)**][8_512] |       [8/768][8_768]        |
  | **L=10** |     [10/128][10_128]      |     [10/256][10_256]      |      [10/512][10_512]       |      [10/768][10_768]       |
  | **L=12** |     [12/128][12_128]      |     [12/256][12_256]      |      [12/512][12_512]       | [**12/768 (Base)**][12_768] |
  
  Here are scores on the devlopment set of six Chinese tasks:
  
  | Model          | Score | book_review | chnsenticorp | lcqmc | tnews(CLUE) | iflytek(CLUE) | ocnli(CLUE) |
  | -------------- | :---: | :----: | :----------: | :---: | :---------: | :-----------: | :---------: |
  | RoBERTa-Tiny   | 72.3  |  83.4  |     91.4     | 81.8  |    62.0     |     55.0      |    60.3     |
  | RoBERTa-Mini   | 75.9  |  85.7  |     93.7     | 86.1  |    63.9     |     58.3      |    67.4     |
  | RoBERTa-Small  | 76.9  |  87.5  |     93.4     | 86.5  |    65.1     |     59.4      |    69.7     |
  | RoBERTa-Medium | 78.0  |  88.7  |     94.8     | 88.1  |    65.6     |     59.5      |    71.2     |
  | RoBERTa-Base   | 79.7  |  90.1  |     95.2     | 89.2  |    67.0     |     60.9      |    75.5     |
  
  For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained with the sequence length of 128:
  
  - epochs: 3, 5, 8
  - batch sizes: 32, 64
  - learning rates: 3e-5, 1e-4, 3e-4
  
  ## How to use
  
  You can use this model directly with a pipeline for masked language modeling (take the case of RoBERTa-Medium):
  
  ```python
  >>> from transformers import pipeline
  >>> unmasker = pipeline('fill-mask', model='uer/chinese_roberta_L-8_H-512')
  >>> unmasker("中国的首都是[MASK]京。")
  [
      {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]', 
       'score': 0.8701988458633423, 
       'token': 1266, 
       'token_str': '北'},
      {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
       'score': 0.1194809079170227, 
       'token': 1298, 
       'token_str': '南'},
      {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]', 
       'score': 0.0037803512532263994, 
       'token': 691, 
       'token_str': '东'},
      {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
       'score': 0.0017127094324678183, 
       'token': 3249,
       'token_str': '普'},
      {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
       'score': 0.001687526935711503,
       'token': 3307, 
       'token_str': '望'}
  ]
  ```
  
  Here is how to use this model to get the features of a given text in PyTorch:
  
  ```python
  from transformers import BertTokenizer, BertModel
  tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
  model = BertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
  text = "用你喜欢的任何文本替换我。"
  encoded_input = tokenizer(text, return_tensors='pt')
  output = model(**encoded_input)
  ```
  
  and in TensorFlow:
  
  ```python
  from transformers import BertTokenizer, TFBertModel
  tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
  model = TFBertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
  text = "用你喜欢的任何文本替换我。"
  encoded_input = tokenizer(text, return_tensors='tf')
  output = model(encoded_input)
  ```
  
  ## Training data
  
  [CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.
  
  ## Training procedure
  
  Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 1