readme: ---
  language: zh
  datasets: CLUECorpusSmall
  widget: 
  - text: "最近一趟去北京的[MASK]几点发车"
  
  
  ---
  
  
  # Chinese word-based RoBERTa Miniatures
  
  ## Model description
  
  This is the set of 5 Chinese word-based RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).
  
  Most Chinese pre-trained weights are based on Chinese character. Compared with character-based models, word-based models are faster (because of shorter sequence length) and have better performance according to our experimental results. To this end, we released the 5 Chinese word-based RoBERTa models of different sizes. In order to facilitate users to reproduce the results, we used the publicly available corpus and word segmentation tool, and provided all training details.
  
  Notice that the output results of Hosted inference API (right) are not properly displayed. When the predicted word has multiple characters, the single word instead of entire sentence is displayed. One can click **JSON Output** for normal output results.
  
  You can download the 5 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:
  
  |          |           Link           |
  | -------- | :-----------------------: |
  | **word-based RoBERTa-Tiny** | [**L=2/H=128 (Tiny)**][2_128] |
  | **word-based RoBERTa-Mini** | [**L=4/H=256 (Mini)**][4_256] |
  | **word-based RoBERTa-Small** | [**L=4/H=512 (Small)**][4_512] |
  | **word-based RoBERTa-Medium** | [**L=8/H=512 (Medium)**][8_512] |
  | **word-based RoBERTa-Base** | [**L=12/H=768 (Base)**][12_768] |
  
  Compared with [char-based models](https://huggingface.co/uer/chinese_roberta_L-2_H-128), word-based models achieve better results in most cases. Here are scores on the devlopment set of six Chinese tasks:
  
  | Model          | Score | book_review | chnsenticorp | lcqmc | tnews(CLUE) | iflytek(CLUE) | ocnli(CLUE) |
  | -------------- | :---: | :----: | :----------: | :---: | :---------: | :-----------: | :---------: |
  | RoBERTa-Tiny(char)       | 72.3            |  83.4      |     91.4         | 81.8      |    62.0         |     55.0          |    60.3         |
  | **RoBERTa-Tiny(word)**   | **74.4(+2.1)**  |  **86.7**  |     **93.2**     | **82.0**  |    **66.4**     |     **58.2**      |    **59.6**     |
  | RoBERTa-Mini(char)       | 75.9            |  85.7      |     93.7         | 86.1      |    63.9         |     58.3          |    67.4         |
  | **RoBERTa-Mini(word)**   | **76.9(+1.0)**  |  **88.5**  |     **94.1**     | **85.4**  |    **66.9**     |     **59.2**      |    **67.3**     |
  | RoBERTa-Small(char)      | 76.9            |  87.5      |     93.4         | 86.5      |    65.1         |     59.4          |    69.7         |
  | **RoBERTa-Small(word)**  | **78.4(+1.5)**  |  **89.7**  |     **94.7**     | **87.4**  |    **67.6**     |     **60.9**      |    **69.8**     |
  | RoBERTa-Medium(char)     | 78.0            |  88.7      |     94.8         | 88.1      |    65.6         |     59.5          |    71.2         |
  | **RoBERTa-Medium(word)** | **79.1(+1.1)**  |  **90.0**  |     **95.1**     | **88.0**  |    **67.8**     |     **60.6**      |    **73.0**     |
  | RoBERTa-Base(char)       | 79.7            |  90.1      |     95.2         | 89.2      |    67.0         |     60.9          |    75.5         |
  | **RoBERTa-Base(word)**   | **80.4(+0.7)**  |  **91.1**  |     **95.7**     | **89.4**  |    **68.0**     |     **61.5**      |    **76.8**     |
  
  For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained with the sequence length of 128:
  
  - epochs: 3, 5, 8
  - batch sizes: 32, 64
  - learning rates: 3e-5, 1e-4, 3e-4
  
  ## How to use
  
  You can use this model directly with a pipeline for masked language modeling (take the case of word-based RoBERTa-Medium):
  
  ```python
  >>> from transformers import pipeline
  >>> unmasker = pipeline('fill-mask', model='uer/roberta-medium-word-chinese-cluecorpussmall')
  >>> unmasker("[MASK]的首都是北京。")
  [
      {'sequence': '中国 的首都是北京。',
       'score': 0.21525809168815613, 
       'token': 2873, 
       'token_str': '中国'}, 
      {'sequence': '北京 的首都是北京。', 
       'score': 0.15194718539714813, 
       'token': 9502, 
       'token_str': '北京'}, 
      {'sequence': '我们 的首都是北京。', 
       'score': 0.08854265511035919, 
       'token': 4215, 
       'token_str': '我们'},
      {'sequence': '美国 的首都是北京。', 
       'score': 0.06808705627918243, 
       'token': 7810, 
       'token_str': '美国'}, 
      {'sequence': '日本 的首都是北京。', 
       'score': 0.06071401759982109, 
       'token': 7788, 
       'token_str': '日本'}
  ]
  ```
  
  Here is how to use this model to get the features of a given text in PyTorch:
  
  ```python
  from transformers import AlbertTokenizer, BertModel
  tokenizer = AlbertTokenizer.from_pretrained('uer/roberta-medium-word-chinese-cluecorpussmall')
  model = BertModel.from_pretrained("uer/roberta-medium-word-chinese-cluecorpussmall")
  text = "用你喜欢的任何文本替换我。"
  encoded_i