readme: # CirBERTa
  ### Apply the Circular to the Pretraining Model
  
  
  
  
  | 预训练模型            | 学习率 | batchsize | 设备   | 语料库 | 时间 | 优化器 |
  | --------------------- | ------ | --------- | ------ | ------ | ---- | ------ |
  | CirBERTa-Chinese-Base | 1e-5   | 256      | 10张3090+3张A100 | 200G   | 2月 | AdamW  |
  
  
  使用通用语料（WuDao 200G) 进行无监督预训练
  
  在多项中文理解任务上，CirBERTa-Base模型超过MacBERT-Chinese-Large/RoBERTa-Chinese-Large
  
  
  ### 加载与使用
  
  依托于huggingface-transformers
  
  ```
  from transformers import AutoTokenizer,AutoModel
  
  tokenizer = AutoTokenizer.from_pretrained("WENGSYX/CirBERTa-Chinese-Base")
  model = AutoModel.from_pretrained("WENGSYX/CirBERTa-Chinese-Base")
  ```
  
  
  ### 引用:
  (暂时先引用这个，论文正在撰写...)
  ```
  @misc{CirBERTa,
    title={CirBERTa: Apply the Circular to the Pretraining Model},
    author={Yixuan Weng},
    howpublished={\url{https://github.com/WENGSYX/CirBERTa}},
    year={2022}
  }
  ```
  