readme: ---
  language:
  - en
  datasets:
  - the_pile
  library_name: nemo
  tags:
  - medical
  ---
  # GatorTron-OG
  
  GatorTron-OG is a 345m-parameter cased Megatron checkpoint from the University of Floria and NVIDIA, pre-trained on a dataset consisting of,
  
    * 82B words of de-identified clinical notes from the University of Florida Health System,
    * 6.1B words from PubMed CC0,
    * 2.5B words from WikiText,
    * 0.5B words from MIMIC-III itself.
  
  The model is designed to provide improved language understanding for downstream clinical tasks. It was trained and is released with a 50K token customized clinical vocabulary trained on the above listed data distribution.
  
  ### More Information
  
  Additional information about the model, including license information, can be found [here](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_og).
  
  Paper here: https://arxiv.org/abs/2203.03540