readme: ---
  language: zh
  tags:
  - structbert
  - pytorch
  - tf2.0
  inference: False
  ---
  
  # StructBERT: Un-Official Copy
  
  Official Repository Link: https://github.com/alibaba/AliceMind/tree/main/StructBERT
  
  **Claimer** 
  * This model card is not produced by [AliceMind Team](https://github.com/alibaba/AliceMind/)
  
  ## Reproduce HFHub models:
  Download model/tokenizer vocab
  ```bash
  wget https://raw.githubusercontent.com/alibaba/AliceMind/main/StructBERT/config/ch_large_bert_config.json && mv ch_large_bert_config.json config.json
  wget https://raw.githubusercontent.com/alibaba/AliceMind/main/StructBERT/config/ch_vocab.txt
  wget https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/StructBERT/ch_model && mv ch_model pytorch_model.bin
  ```
  
  ```python
  from transformers import BertConfig, BertModel, BertTokenizer
  config = BertConfig.from_pretrained("./config.json")
  model = BertModel.from_pretrained("./", config=config)
  tokenizer = BertTokenizer.from_pretrained("./")
  model.push_to_hub("structbert-large-zh")
  tokenizer.push_to_hub("structbert-large-zh")
  ```
  
  [https://arxiv.org/abs/1908.04577](https://arxiv.org/abs/1908.04577)
  
  # StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
  ## Introduction
  We extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. 
  Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential 
  order of words and sentences, which leverage language structures at the word and sentence levels, 
  respectively.
  ## Pre-trained models
  |Model | Description | #params | Download |
  |------------------------|-------------------------------------------|------|------|
  |structbert.en.large | StructBERT using the BERT-large architecture | 340M | [structbert.en.large](https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/StructBERT/en_model) |
  |structroberta.en.large | StructRoBERTa continue training from RoBERTa | 355M | Coming soon |
  |structbert.ch.large | Chinese StructBERT; BERT-large architecture | 330M | [structbert.ch.large](https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/StructBERT/ch_model) |
  
  ## Results
  The results of GLUE & CLUE tasks can be reproduced using the hyperparameters listed in the following "Example usage" section.
  #### structbert.en.large
  [GLUE benchmark](https://gluebenchmark.com/leaderboard)
  
  |Model| MNLI | QNLIv2 | QQP | SST-2 | MRPC | 
  |--------------------|-------|-------|-------|-------|-------|
  |structbert.en.large |86.86% |93.04% |91.67% |93.23% |86.51% |
  #### structbert.ch.large
  [CLUE benchmark](https://www.cluebenchmarks.com/)
  
  |Model | CMNLI | OCNLI | TNEWS | AFQMC |
  |--------------------|-------|-------|-------|-------|
  |structbert.ch.large |84.47% |81.28% |68.67% |76.11% | 
  
  ## Example usage
  #### Requirements and Installation
  * [PyTorch](https://pytorch.org/) version >= 1.0.1
  
  * Install other libraries via
  ```
  pip install -r requirements.txt
  ```
  
  * For faster training install NVIDIA's [apex](https://github.com/NVIDIA/apex) library
  
  #### Finetune MNLI
  
  ```
  python run_classifier_multi_task.py \
    --task_name MNLI \
    --do_train \
    --do_eval \
    --do_test \
    --amp_type O1 \
    --lr_decay_factor 1 \
    --dropout 0.1 \
    --do_lower_case \
    --detach_index -1 \
    --core_encoder bert \
    --data_dir path_to_glue_data \
    --vocab_file config/vocab.txt \
    --bert_config_file config/large_bert_config.json \
    --init_checkpoint path_to_pretrained_model \
    --max_seq_length 128 \
    --train_batch_size 32 \
    --learning_rate 2e-5 \
    --num_train_epochs 3 \
    --fast_train \
    --gradient_accumulation_steps 1 \
    --output_dir path_to_output_dir 
  ```
  
  ## Citation
  If you use our work, please cite:
  ```
  @article{wang2019structbert,
    title={Structbert: Incorporating language structures into pre-training for deep language understanding},
    author={Wang, Wei and Bi, Bin and Yan, Ming and Wu, Chen and Bao, Zuyi and Xia, Jiangnan and Peng, Liwei and Si, Luo},
    journal={arXiv preprint arXiv:1908.04577},
    year={2019}
  }
  ```