readme: ---
  language: 
    - zh
  
  license: apache-2.0
  
  tags:
    - classification
  
  inference: false
  
  ---
  
  # IDEA-CCNL/Erlangshen-TCBert-110M-Sentence-Embedding-Chinese
  
  - Github: [Fengshenbang-LM](https://github.com/IDEA-CCNL/Fengshenbang-LM)
  - Docs: [Fengshenbang-Docs](https://fengshenbang-doc.readthedocs.io/)
  
  ## 简介 Brief Introduction
  
  110M参数的句子表征Topic Classification BERT (TCBert)。
  
  The TCBert with 110M parameters is pre-trained for sentence representation for Chinese topic classification tasks.
  
  ## 模型分类 Model Taxonomy
  
  |  需求 Demand  | 任务 Task       | 系列 Series      | 模型 Model    | 参数 Parameter | 额外 Extra |
  |  :----:  | :----:  | :----:  | :----:  | :----:  | :----:  |
  | 通用 General  | 句子表征 | 二郎神 Erlangshen | TCBert (sentence representation)      |      110M     |   Chinese     |
  
  ## 模型信息 Model Information
  
  
  为了提高模型在话题分类上句子表征效果，我们收集了大量话题分类数据进行基于prompts的对比学习预训练。
  
  To improve the model performance on sentence representation for the topic classification task, we collected numerous topic classification datasets for contrastive pre-training based on general prompts.
  ### 下游效果 Performance
  
  我们为每个数据集设计了两个prompt模板。
  
  We customize two prompts templates for each dataset.
  
  第一个prompt模板：
  
  For ***prompt template 1***:
  
  | Dataset |     Prompt template 1    |
  |---------|:------------------------:|
  | TNEWS   | 下面是一则关于__的新闻： |
  | CSLDCP  | 这一句描述__的内容如下： |
  | IFLYTEK | 这一句描述__的内容如下： |
  
  
  第一个prompt模板的微调实验结果：
  
  The **fine-tuning** results for prompt template 1:
  
  | Model           |  TNEWS | CLSDCP | IFLYTEK |
  |-----------------|:------:|:------:|:-------:|
  | Macbert-base    | 55.02  | 57.37  |  51.34  |
  | Macbert-large   | 55.77  | 58.99  |  50.31  |
  | Erlangshen-1.3B | 57.36  | 62.35  |  53.23  |
  | TCBert-base<sub>110M-Classification-Chinese     | 55.57  | 58.60  |  49.63  |
  | TCBert-large<sub>330M-Classification-Chinese    | 56.17  | 60.06  |  51.34  |
  | TCBert-1.3B<sub>1.3B-Classification-Chinese     | 57.41  | 65.10  |  53.75  |
  | TCBert-base<sub>110M-Sentence-Embedding-Chinese     | 54.68  | 59.78  |  49.40  |
  | TCBert-large<sub>330M-Sentence-Embedding-Chinese    | 55.32  | 62.07  |  51.11  |
  | TCBert-1.3B<sub>1.3B-Sentence-Embedding-Chinese     | 57.46  | 65.04  |  53.06  |
  
  
  第一个prompt模板的句子相似度结果：
  
  The **sentence similarity** results for prompt template 1:
  
  |                 |   TNEWS  |           |   CSLDCP  |           |  IFLYTEK  |           |
  |-----------------|:--------:|:---------:|:---------:|:---------:|:---------:|:---------:|
  | Model           | referece | whitening | reference | whitening | reference | whitening |
  | Macbert-base    |  43.53   |   47.16   |   33.50   |   36.53   |   28.99   |   33.85   |
  | Macbert-large   |  46.17   |   49.35   |   37.65   |   39.38   |   32.36   |   35.33   |
  | Erlangshen-1.3B |  45.72   |   49.60   |   40.56   |   44.26   |   29.33   |   36.48   |
  | TCBert-base<sub>110M-Classification-Chinese     |  48.61   |   51.99   |   43.31   |   45.15   |   33.45   |   37.28   |
  | TCBert-large<sub>330M-Classification-Chinese    |  50.50   |   52.79   |   52.89   |   53.89   |   34.93   |   38.31   |
  | TCBert-1.3B<sub>1.3B-Classification-Chinese     |  50.80   |   51.59   |   51.93   |   54.12   |   33.96   |   38.08   |
  | TCBert-base<sub>110M-Sentence-Embedding-Chinese     |  45.82   |   47.06   |   42.91   |   43.87   |   33.28   |   34.76   |
  | TCBert-large<sub>330M-Sentence-Embedding-Chinese    |  50.10   |   50.90   |   53.78   |   53.33   |   37.62   |   36.94   |
  | TCBert-1.3B<sub>1.3B-Sentence-Embedding-Chinese     |  50.70   |   53.48   |   52.66   |   54.40   |   36.88   |   38.48   |
  
  第二个prompt模板：
  
  For ***prompt template 2***:
  | Dataset |     Prompt template 2    |
  |---------|:------------------------:|
  | TNEWS   | 接下来的新闻，是跟__相关的内容： |
  | CSLDCP  | 接下来的学科，是跟__相关： |
  | IFLYTEK |  接下来的生活内容，是跟__相关： |
  
  第二个prompt模板的微调结果：
  
  The **fine-tuning** results for prompt template 2:
  
  | Model           |  TNEWS | CLSDCP | IFLYTEK |
  |-----------------|:------:|:------:|:-------:|
  | Macbert-base    | 54.78  | 58.38  |  50.83  |
  | Macbert-large   | 56.77  | 60.22  |  51.63  |
  | Erlangshen-1.3B | 57.81  | 62.80  |  52.77  |
  | TCBert-base<sub>110M-Classification-Chinese     | 54.58  | 59.16  |  49.80  |
  | TCBert-large<sub>330M-Classification-Chinese    | 56.22  | 61.23  |  50.77  |
  | TCBert-1.3B<sub>1.3B-Classification-Chinese      | 57.41  | 64.82  |  53.34  |
  | TCBert-base<sub>110M-Sentence-Embedding-Chinese      | 54.68  | 59.78  |  49.40  |
  | TCBert-large<sub>330M-Sentence-Embedding-Chinese    | 55.32  | 62.07  |  51.11  |
  | TCBert-1.3B<sub>1.3B-Sentence-Embedding-Chinese     | 56.87  | 65.83  |  52.94  |
  
  
  第二个prompt模板的句子相似度结果：
  
  The **sentence similarity** results for prompt template 2:
  
  |                 |   TNEWS  |           |   CSLDCP  |           |  IFLYTEK  |           |
  |-----------------|:--------:|:---------:|:---------:|:---------:|:---------:|:---------:|
  | Model           | referece | whitening | reference | whitening | reference | whitening |
  | Macbert-base    |  42.29   |   45.22   |   34.23   |   37.48   