KEPTlongfomer is a medical knowledge enhanced version of Longformer that was further pre-trained using contrastive learning. The model achieves SOTA performance on auto ICD coding on MIMIC-III as of 11/12/2022. A sister model for better performance is available here.
Pre-training
We initialized this model from clinical longformer.
And then pretrained with Hierarchical Self-Alignment Pretrain (HSAP) using Knowledge Graph UMLS. This includes (a) Hierarchy, (b) Synonym, (c) Abbreviation. For more info, see section 3.3 in paper. The learning rate was 5e-5, weight decay was 0.01, adam epsilon was 1e-5.
Usage
See our github for how to use this with prompts on auto ICD coding.
With the following result:
Metric | Score |
---|---|
rec_micro | =0.5729403619819988 |
rec_macro | =0.11342156911120573 |
rec_at_8 | =0.4094837705486378 |
rec_at_75 | =0.8470734920535119 |
rec_at_50 | =0.8005338782352 |
rec_at_5 | =0.2891628170355805 |
rec_at_15 | =0.5768778119750537 |
prec_micro | =0.6411968713105065 |
prec_macro | =0.12227610414493029 |
prec_at_8 | =0.7760972716488731 |
prec_at_75 | =0.197504942665085 |
prec_at_50 | =0.2768090154211151 |
prec_at_5 | =0.8483392645314354 |
prec_at_15 | =0.6178529062870699 |
f1_micro | =0.6051499904242899 |
f1_macro | =0.11768251595637802 |
f1_at_8 | =0.536107150495997 |
f1_at_75 | =0.32032290907137506 |
f1_at_50 | =0.411373195944102 |
f1_at_5 | =0.43131028155283435 |
f1_at_15 | =0.5966627077602488 |
auc_micro | =0.9651754312635265 |
auc_macro | =0.8566590059725866 |
acc_micro | =0.43384592341105344 |
acc_macro | =0.08639139221100567 |
- Downloads last month
- 74
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.