title: Has Your Pretrained Model Improved? A Multi-head Posterior Based
    Approach
summary: The emergence of pretrained models has significantly impacted from Natural
  Language Processing (NLP) and Computer Vision to relational datasets.
  Traditionally, these models are assessed through fine-tuned downstream tasks.
  However, this raises the question of how to evaluate these models more
  efficiently and more effectively. In this study, we explore a novel approach
  where we leverage the meta features associated with each entity as a source of
  worldly knowledge and employ entity representations from the models. We propose
  using the consistency between these representations and the meta features as a
  metric for evaluating pretrained models. Our method's effectiveness is
  demonstrated across various domains, including models with relational datasets,
  large language models and images models.
paper_authors._id: 659cbe46442979361cd46580
paper_authors.name: Prince Aboagye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T09:07:54.590Z
paper_authors._id: 60db7be9bf02367a70af5bb9
paper_authors.avatarUrl: /avatars/6bddc3e0d04775aaff9fe71c5f237baa.svg
paper_authors.fullname: Prince Aboagye
paper_authors.type: user
paper_authors.user: paboagye
paper_authors._id: 659cbe46442979361cd46581
paper_authors.name: Yan Zheng
paper_authors._id: 659cbe46442979361cd46582
paper_authors.name: Junpeng Wang
paper_authors._id: 659cbe46442979361cd46583
paper_authors.name: Uday Singh Saini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T09:09:31.635Z
paper_authors._id: 634ee7a64677a5891c0d6cba
paper_authors.avatarUrl: /avatars/df548841cbf388f1de2f0245fa818fef.svg
paper_authors.fullname: Uday Singh Saini
paper_authors.type: user
paper_authors.user: lazypikachu23
paper_authors._id: 659cbe46442979361cd46584
paper_authors.name: Xin Dai
paper_authors._id: 659cbe46442979361cd46585
paper_authors.name: Michael Yeh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T09:10:12.679Z
paper_authors._id: 646dbc8b212146121301212c
paper_authors.avatarUrl: /avatars/bffd1cc6e4920422ed577d4421057ca3.svg
paper_authors.fullname: Michael Yeh
paper_authors.type: user
paper_authors.user: NCGTrigger
paper_authors._id: 659cbe46442979361cd46586
paper_authors.name: Yujie Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T09:10:22.966Z
paper_authors._id: 6267b82333e1845bf1fe5d7a
paper_authors.avatarUrl: /avatars/43d3b12c4df7db2f3fadc2c911a59908.svg
paper_authors.fullname: fanyujie
paper_authors.type: user
paper_authors.user: fanyujie
paper_authors._id: 659cbe46442979361cd46587
paper_authors.name: Zhongfang Zhuang
paper_authors._id: 659cbe46442979361cd46588
paper_authors.name: Shubham Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-10T08:38:55.246Z
paper_authors._id: 64ef7b470af6d9bfbc8b976f
paper_authors.avatarUrl: /avatars/8be23c2f74f94ee2aa2ffa78dddaf2e9.svg
paper_authors.fullname: shubham jain
paper_authors.type: user
paper_authors.user: shubhjai
paper_authors._id: 659cbe46442979361cd46589
paper_authors.name: Liang Wang
paper_authors._id: 659cbe46442979361cd4658a
paper_authors.name: Wei Zhang
paper_title: Has Your Pretrained Model Improved? A Multi-head Posterior Based
    Approach
paper_summary: The emergence of pretrained models has significantly impacted from Natural
  Language Processing (NLP) and Computer Vision to relational datasets.
  Traditionally, these models are assessed through fine-tuned downstream tasks.
  However, this raises the question of how to evaluate these models more
  efficiently and more effectively. In this study, we explore a novel approach
  where we leverage the meta features associated with each entity as a source of
  worldly knowledge and employ entity representations from the models. We propose
  using the consistency between these representations and the meta features as a
  metric for evaluating pretrained models. Our method's effectiveness is
  demonstrated across various domains, including models with relational datasets,
  large language models and images models.
paper_ai_keywords: pretrained models NLP Computer Vision relational datasets fine-tuned downstream tasks meta features entity representations consistency relational datasets large language models images models