readme: ---
  license: mit
  widget:
  language:
  - en
  
  datasets:
  - pytorrent
  ---
  
  #  🔥 RoBERTa-MLM-based PyTorrent 1M 🔥
  Pretrained weights based on [PyTorrent Dataset](https://github.com/fla-sil/PyTorrent) which is a curated data from a large official Python packages.
  We use PyTorrent dataset to train a preliminary DistilBERT-Masked Language Modeling(MLM) model from scratch. The trained model, along with the dataset, aims to help researchers to easily and efficiently work on a large dataset of Python packages using only 5 lines of codes to load the transformer-based model. We use 1M raw Python scripts of PyTorrent that includes 12,350,000 LOC to train the model. We also train a byte-level Byte-pair encoding (BPE) tokenizer that includes 56,000 tokens, which is truncated LOC with the length of 50 to save computation resources. 
  
  ### Training Objective
  This model is trained with a Masked Language Model (MLM) objective.
  
  ## How to use the model?
  ```python
  from transformers import AutoTokenizer, AutoModel
  
  
  tokenizer = AutoTokenizer.from_pretrained("Fujitsu/pytorrent")
  model = AutoModel.from_pretrained("Fujitsu/pytorrent")
  ```
  ## Citation
  Preprint: [https://arxiv.org/pdf/2110.01710.pdf](https://arxiv.org/pdf/2110.01710.pdf)
  ```
  @misc{bahrami2021pytorrent,
        title={PyTorrent: A Python Library Corpus for Large-scale Language Models}, 
        author={Mehdi Bahrami and N. C. Shrikanth and Shade Ruangwan and Lei Liu and Yuji Mizobuchi and Masahiro Fukuyori and Wei-Peng Chen and Kazuki Munakata and Tim Menzies},
        year={2021},
        eprint={2110.01710},
        archivePrefix={arXiv},
        primaryClass={cs.SE},
        howpublished={https://arxiv.org/pdf/2110.01710},
  }
  ```
  