readme: ---
  license: apache-2.0
  language: "en"
  tags:
  - longformer
  - clinical
  - biomedical
  ---
  
  <span style="font-size:larger;">**KEPTlongfomer**</span> is a medical knowledge enhanced version of Longformer that was further pre-trained using [contrastive learning](https://arxiv.org/pdf/2210.03304.pdf). 
  The model achieves SOTA performance on auto ICD coding on MIMIC-III as of 11/12/2022.
  A sister model for better performance is available [here](https://huggingface.co/whaleloops/KEPTlongformer-PMM3/).
  
  ### Pre-training
  We initialized this model from [clinical longformer](https://huggingface.co/yikuan8/Clinical-Longformer).
  
  And then pretrained with Hierarchical Self-Alignment Pretrain (HSAP) using Knowledge Graph UMLS.
  This includes (a) Hierarchy, (b) Synonym, (c) Abbreviation. For more info, see section 3.3 in [paper](https://arxiv.org/pdf/2210.03304.pdf).
  The learning rate was 5e-5, weight decay was 0.01, adam epsilon was 1e-5.
  
  ### Usage
  See our [github](https://github.com/whaleloops/KEPT/tree/rerank300) for how to use this with prompts on auto ICD coding.
  
  With the following result:
  | Metric  | Score |
  | ------------- | ------------- |
  |rec_micro| =0.5729403619819988|
  |rec_macro| =0.11342156911120573|
  |rec_at_8| =0.4094837705486378|
  |rec_at_75| =0.8470734920535119|
  |rec_at_50| =0.8005338782352|
  |rec_at_5| =0.2891628170355805|
  |rec_at_15| =0.5768778119750537|
  |prec_micro| =0.6411968713105065|
  |prec_macro| =0.12227610414493029|
  |prec_at_8| =0.7760972716488731|
  |prec_at_75| =0.197504942665085|
  |prec_at_50| =0.2768090154211151|
  |prec_at_5| =0.8483392645314354|
  |prec_at_15| =0.6178529062870699|
  |f1_micro| =0.6051499904242899|
  |f1_macro| =0.11768251595637802|
  |f1_at_8| =0.536107150495997|
  |f1_at_75| =0.32032290907137506|
  |f1_at_50| =0.411373195944102|
  |f1_at_5| =0.43131028155283435|
  |f1_at_15| =0.5966627077602488|
  |auc_micro| =0.9651754312635265|
  |auc_macro| =0.8566590059725866|
  |acc_micro| =0.43384592341105344|
  |acc_macro| =0.08639139221100567|
  
  
  
  