readme: ---
  license: mit
  
  datasets:
    - damlab/uniprot
  metrics:
    - accuracy
  
  widget:
   - text: 'involved_in GO:0006468 involved_in GO:0007165 located_in GO:0042470 involved_in GO:0070372'
     example_title: 'Function'
  
  ---
  
  # GO-Language model
  
  ## Table of Contents
  - [Summary](#model-summary)
  - [Model Description](#model-description)
  - [Intended Uses & Limitations](#intended-uses-&-limitations)
  - [How to Use](#how-to-use)
  - [Training Data](#training-data)
  - [Training Procedure](#training-procedure)
    - [Preprocessing](#preprocessing)
    - [Training](#training)
  - [Evaluation Results](#evaluation-results)
  - [BibTeX Entry and Citation Info](#bibtex-entry-and-citation-info)
  
  ## Summary
  
  This model was built as a way to encode the Gene Ontology definition of a protein as vector representation.
  It was trained on a collection of gene-ontology terms from model organisms.
  Each function was sorted by the ID number and combined with its annotation description ie (`is_a`, `enables`, `located_in`, etc).
  The model is tokenized such that each description and GO term is its own token.
  This is intended to be used as a translation model between PROT-BERT and GO-Language.
  That type of translation model will be useful for predicting the function of novel genes.
  
  ## Model Description
  
  This model was trained using the damlab/uniprot dataset on the `go` field with 256 token chunks and a 15% mask rate.
  
  
  ## Intended Uses & Limitations
  
  This model is a useful encapsulation of gene ontology functions.
  It allows both an exploration of gene-level similarities as well as comparisons between functional terms.
  
  ## How to use
  
  As this is a BERT-style Masked Language learner, it can be used to determine the most likely token a masked position.
  
  ```python
  from transformers import pipeline
  
  unmasker = pipeline("fill-mask", model="damlab/GO-language")
  
  unmasker("involved_in [MASK] involved_in GO:0007165 located_in GO:0042470 involved_in GO:0070372")
  
  [{'score': 0.1040298342704773,
    'token': 103,
    'token_str': 'GO:0002250',
    'sequence': 'involved_in GO:0002250 involved_in GO:0007165 located_in GO:0042470 involved_in GO:0070372'},
   {'score': 0.018045395612716675,
    'token': 21,
    'token_str': 'GO:0005576',
    'sequence': 'involved_in GO:0005576 involved_in GO:0007165 located_in GO:0042470 involved_in GO:0070372'},
   {'score': 0.015035462565720081,
    'token': 50,
    'token_str': 'GO:0000139',
    'sequence': 'involved_in GO:0000139 involved_in GO:0007165 located_in GO:0042470 involved_in GO:0070372'},
   {'score': 0.01181247178465128,
    'token': 37,
    'token_str': 'GO:0007165',
    'sequence': 'involved_in GO:0007165 involved_in GO:0007165 located_in GO:0042470 involved_in GO:0070372'},
   {'score': 0.01000668853521347,
    'token': 14,
    'token_str': 'GO:0005737',
    'sequence': 'involved_in GO:0005737 involved_in GO:0007165 located_in GO:0042470 involved_in GO:0070372'}
  ]
  
  ```
  
  ## Training Data
  
  The dataset was trained using [damlab/uniprot](https://huggingface.co/datasets/damlab/uniprot) from a random initial model.
  The Gene Ontology functions were sorted (by ID number) along with annotating term.
  
  ## Training Procedure
  
  ### Preprocessing
  
  All strings were concatenated and chunked into 256 token chunks for training. A random 20% of chunks were held for validation.
  
  ### Training
  
  Training was performed with the HuggingFace training module using the MaskedLM data loader with a 15% masking rate. The learning rate was set at E-5, 50K warm-up steps, and a cosine_with_restarts learning rate schedule and continued until 3 consecutive epochs did not improve the loss on the held-out dataset.
  
  
  ## BibTeX Entry and Citation Info
  
  [More Information Needed]
  