Edit model card

KEPTlongfomer is a medical knowledge enhanced version of Longformer that was further pre-trained using contrastive learning. The model achieves SOTA performance on auto ICD coding on MIMIC-III as of 11/12/2022. A sister model for better performance is available here.

Pre-training

We initialized this model from clinical longformer.

And then pretrained with Hierarchical Self-Alignment Pretrain (HSAP) using Knowledge Graph UMLS. This includes (a) Hierarchy, (b) Synonym, (c) Abbreviation. For more info, see section 3.3 in paper. The learning rate was 5e-5, weight decay was 0.01, adam epsilon was 1e-5.

Usage

See our github for how to use this with prompts on auto ICD coding.

With the following result:

Metric Score
rec_micro =0.5729403619819988
rec_macro =0.11342156911120573
rec_at_8 =0.4094837705486378
rec_at_75 =0.8470734920535119
rec_at_50 =0.8005338782352
rec_at_5 =0.2891628170355805
rec_at_15 =0.5768778119750537
prec_micro =0.6411968713105065
prec_macro =0.12227610414493029
prec_at_8 =0.7760972716488731
prec_at_75 =0.197504942665085
prec_at_50 =0.2768090154211151
prec_at_5 =0.8483392645314354
prec_at_15 =0.6178529062870699
f1_micro =0.6051499904242899
f1_macro =0.11768251595637802
f1_at_8 =0.536107150495997
f1_at_75 =0.32032290907137506
f1_at_50 =0.411373195944102
f1_at_5 =0.43131028155283435
f1_at_15 =0.5966627077602488
auc_micro =0.9651754312635265
auc_macro =0.8566590059725866
acc_micro =0.43384592341105344
acc_macro =0.08639139221100567
Downloads last month
74
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.