readme: # LIMIT-BERT
  
  Code and model for the *EMNLP 2020 Findings* paper: 
  
  [LIMIT-BERT: Linguistic Informed Multi-task BERT](https://arxiv.org/abs/1910.14296)) 
  
  ## Contents
  
  1. [Requirements](#Requirements)
  2. [Training](#Training)
  
  ## Requirements
  
  * Python 3.6 or higher.
  * Cython 0.25.2 or any compatible version.
  * [PyTorch](http://pytorch.org/) 1.0.0+. 
  * [EVALB](http://nlp.cs.nyu.edu/evalb/). Before starting, run `make` inside the `EVALB/` directory to compile an `evalb` executable. This will be called from Python for evaluation. 
  * [pytorch-transformers](https://github.com/huggingface/pytorch-transformers) PyTorch 1.0.0+ or any compatible version.
  
  #### Pre-trained Models (PyTorch)
  The following pre-trained models are available for download from Google Drive:
  * [`LIMIT-BERT`](https://drive.google.com/open?id=1fm0cK2A91iLG3lCpwowCCQSALnWS2X4i): 
    PyTorch version, same setting with BERT-Large-WWM，loading model with [pytorch-transformers](https://github.com/huggingface/pytorch-transformers).
  
  ## How to use
  
  ```
  from transformers import AutoTokenizer, AutoModel
  tokenizer = AutoTokenizer.from_pretrained("cooelf/limitbert")
  model = AutoModel.from_pretrained("cooelf/limitbert")
  ```
  
  Please see our original repo for the training scripts.
  
  https://github.com/cooelf/LIMIT-BERT
  
  ## Training
  
  To train LIMIT-BERT, simply run:
  ```
  sh run_limitbert.sh
  ```
  ### Evaluation Instructions
  
  To test after setting model path:
  ```
  sh test_bert.sh
  ```
  
  ## Citation
  
  ```
  @article{zhou2019limit,
    title={{LIMIT-BERT}: Linguistic informed multi-task {BERT}},
    author={Zhou, Junru and Zhang, Zhuosheng and Zhao, Hai},
    journal={arXiv preprint arXiv:1910.14296},
    year={2019}
  }
  ```