readme: This is a `distilgpt2` model, finetuned on the Wikitext-103 dataset.
  
  It achieves a perplexity of **18.25** using a "sliding window" context, using the `run_clm.py` script at [https://github.com/neulab/knn-transformers](https://github.com/neulab/knn-transformers).
  
  | Base LM:        | `distilgpt2` | `gpt2` | 
  | :---            |    ----:   |     ---: |
  | base perplexity | 18.25      | 14.84    |
  | + kNN-LM          |  15.03     |   12.57  |
  | + RetoMaton       | **14.70**  |  **12.46**    |
  
  This model was released as part of the paper ["Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval"](https://arxiv.org/pdf/2201.12431.pdf) (ICML'2022).
  
  For more information, see: [https://github.com/neulab/knn-transformers](https://github.com/neulab/knn-transformers)
  
  If you use this model, please cite:
  ```
  @inproceedings{alon2022neuro,
    title={Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval},
    author={Alon, Uri and Xu, Frank and He, Junxian and Sengupta, Sudipta and Roth, Dan and Neubig, Graham},
    booktitle={International Conference on Machine Learning},
    pages={468--485},
    year={2022},
    organization={PMLR}
  }
  ```