readme: ---
  language: 
  - zh
  tags:
  - cpm
  license: mit
  datasets:
  - 100GB Chinese corpus
  ---
  # CPM-Generate
  
  ## Model description
  
  CPM (Chinese Pre-trained Language Model) is a Transformer-based autoregressive language model, with 2.6 billion parameters and 100GB Chinese training data. To the best of our knowledge, CPM is the largest Chinese pre-trained language model, which could facilitate downstream Chinese NLP tasks, such as conversation, essay generation, cloze test, and language understanding. [[Project](https://cpm.baai.ac.cn)] [[Model](https://cpm.baai.ac.cn/download.html)] [[Paper](https://arxiv.org/abs/2012.00413)]
  
  ## Intended uses & limitations
  
  #### How to use
  
  ```python
  from transformers import TextGenerationPipeline, AutoTokenizer, AutoModelWithLMHead
  
  tokenizer = AutoTokenizer.from_pretrained("TsinghuaAI/CPM-Generate")
  model = AutoModelWithLMHead.from_pretrained("TsinghuaAI/CPM-Generate")
  
  text_generator = TextGenerationPipeline(model, tokenizer)
  text_generator('清华大学', max_length=50, do_sample=True, top_p=0.9)
  ```
  
  #### Limitations and bias
  
  The text generated by CPM is automatically generated by a neural network model trained on a large number of texts, which does not represent the authors' or their institutes' official attitudes and preferences. The text generated by CPM is only used for technical and scientific purposes. If it infringes on your rights and interests or violates social morality, please do not propagate it, but contact the authors and the authors will deal with it promptly.
  
  ## Training data
  
  We collect different kinds of texts in our pre-training, including encyclopedia, news, novels, and Q\&A. The details of our training data are shown as follows.
  
  | Data Source | Encyclopedia | Webpage | Story | News  | Dialog |
  | ----------- | ------------ | ------- | ----- | ----- | ------ |
  | **Size**    | ~40GB        | ~39GB   | ~10GB | ~10GB | ~1GB   |
  
  ## Training procedure
  
  Based on the hyper-parameter searching on the learning rate and batch size, we set the learning rate as \\(1.5\times10^{-4}\\) and the batch size as \\(3,072\\), which makes the model training more stable. In the first version, we still adopt the dense attention and the max sequence length is \\(1,024\\). We will implement sparse attention in the future. We pre-train our model for \\(20,000\\) steps, and the first \\(5,000\\) steps are for warm-up. The optimizer is Adam. It takes two weeks to train our largest model using \\(64\\) NVIDIA V100.
  
  ## Eval results
  
  |            | n_param | n_layers | d_model | n_heads | d_head |
  |------------|-------------------:|--------------------:|-------------------:|-------------------:|------------------:|
  | CPM-Small  |               109M |                  12 |                768 |                 12 |                64 |
  | CPM-Medium |               334M |                  24 |              1,024 |                 16 |                64 |
  | CPM-Large  |               2.6B |                  32 |              2,560 |                 32 |                80 |
  
  We evaluate CPM with different numbers of parameters (the details are shown above) on various Chinese NLP tasks in the few-shot (even zero-shot) settings. With the increase of parameters, CPM performs better on most datasets, indicating that larger models are more proficient at language generation and language understanding. We provide results of text classification, chinese idiom cloze test, and short text conversation generation as follows. Please refer to our [paper](https://arxiv.org/abs/2012.00413) for more detailed results.
  
  
  ### Zero-shot performance on text classification tasks
  
  |            |     TNEWS      |    IFLYTEK     |     OCNLI      |
  | ---------- | :------------: | :------------: | :------------: |
  | CPM-Small  |     0.626      |     0.584      |     0.378      |
  | CPM-Medium |     0.618      |     0.635      |     0.379      |
  | CPM-Large  | **0.703** | **0.708** | **0.442** |
  
  ### Performance on Chinese Idiom Cloze (ChID) dataset
  |            |   Supervised   |  Unsupervised  |
  |------------|:--------------:|:--------------:|
  | CPM-Small  |      0.657     |      0.433     |
  | CPM-Medium |      0.695     |      0.524     |
  | CPM-Large  | **0.804** | **0.685** |
  
  ### Performance on Short Text Conversation Generation (STC) dataset
  |                                  |     Average    |     Extrema    |     Greedy     |              Dist-1             |              Dist-2              |
  |----------------------------------|:--------------:|:--------------:|:--------------:|:-------------------------------:|:--------------------------------:|
  | *Few-shot (Unsupervised)* |                |                |                |                                 |                                  |
  | CDial-GPT                        |      0.899     |      0.797     |      0.810     |      1,963 / **0.011**     |          20,814 / 0.126          |
  | CPM-Large                        | **0.928** | **0.805** | **0.815** |      *