readme: ---
  language: en
  tags:
  - longformer
  - cdlm
  license: apache-2.0
  inference: false
  
  ---
  
  
  # Cross-Document Language Modeling
  
  CDLM: Cross-Document Language Modeling. 
  Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan and Ido Dagan. In EMNLP Findings, 2021. [PDF](https://arxiv.org/pdf/2101.00406.pdf)
  
  
  Please note that during our pretraining we used the document and sentence separators, which you might want to add to your data. The document and sentence separators are `<doc-s>`, `</doc-s>` (the last two tokens in the vocabulary), and `<s>`, `</s>`, respectively.
  
  
  ```python
  from transformers import AutoTokenizer, AutoModel
  # load model and tokenizer
  tokenizer = AutoTokenizer.from_pretrained('biu-nlp/cdlm')
  model = AutoModel.from_pretrained('biu-nlp/cdlm')
  ```
  
  The original repo is [here](https://github.com/aviclu/CDLM).
  
  If you find our work useful, please cite the paper as:
  
  ```python
  @article{caciularu2021cross,
    title={Cross-Document Language Modeling},
    author={Caciularu, Avi and Cohan, Arman and Beltagy, Iz and Peters, Matthew E and Cattan, Arie and Dagan, Ido},
    journal={Findings of the Association for Computational Linguistics: EMNLP 2021},
    year={2021}
  }
  ```