readme: ---
  language: protein
  tags:
  - protein
  datasets:
  - uniref-100
  ---
  
  # RITA-M
  
  RITA is a family of autoregressive protein models, developed by a collaboration of [Lighton](https://lighton.ai/), the [OATML group](https://oatml.cs.ox.ac.uk/) at Oxford, and the [Debbie Marks Lab](https://www.deboramarkslab.com/) at Harvard.
  
  
  
  Model | #Params | d_model | layers | lm loss uniref-100
  --- | --- | --- | --- | --- | 
  [Small](https://huggingface.co/lightonai/RITA_s) | 85M  | 768 | 12 | 2.31
  [**Medium**](https://huggingface.co/lightonai/RITA_m) | 300M | 1024 | 24 | 2.01
  [Large](https://huggingface.co/lightonai/RITA_l)| 680M | 1536 | 24 | 1.82
  [XLarge](https://huggingface.co/lightonai/RITA_xl)| 1.2B | 2048 | 24 | 1.70 
  
  
  For full results see our preprint: https://arxiv.org/abs/2205.05789
  
  ## Usage 
  Instantiate a model like so:
  ``` python
  from transformers import AutoModel, AutoModelForCausalLM
  model = AutoModelForCausalLM.from_pretrained("lightonai/RITA_m, trust_remote_code=True")
  tokenizer = AutoTokenizer.from_pretrained("lightonai/RITA_m")
  ```
  for generation we support pipelines:
  ``` python
  from transformers import pipeline
  rita_gen = pipeline('text-generation', model=model, tokenizer=tokenizer)
  sequences = rita_gen("MAB", max_length=20, do_sample=True, top_k=950, repetition_penalty=1.2, 
                       num_return_sequences=2, eos_token_id=2)
  for seq in sequences:
      print(f"seq: {seq['generated_text'].replace(' ', '')}")
  ```
  
  ## How to cite    
  
      @article{hesslow2022rita,
        title={RITA: a Study on Scaling Up Generative Protein Sequence Models},
        author={Hesslow, Daniel and Zanichelli, Niccol{\'o} and Notin, Pascal and Poli, Iacopo and Marks, Debora},
        journal={arXiv preprint arXiv:2205.05789},
        year={2022}
      }