title: Tell me why: Visual foundation models as self-explainable classifiers
summary: Visual foundation models (VFMs) have become increasingly popular due to their
  state-of-the-art performance. However, interpretability remains crucial for
  critical applications. In this sense, self-explainable models (SEM) aim to
  provide interpretable classifiers that decompose predictions into a weighted
  sum of interpretable concepts. Despite their promise, recent studies have shown
  that these explanations often lack faithfulness. In this work, we combine VFMs
  with a novel prototypical architecture and specialized training objectives. By
  training only a lightweight head (approximately 1M parameters) on top of frozen
  VFMs, our approach (ProtoFM) offers an efficient and interpretable solution.
  Evaluations demonstrate that our approach achieves competitive classification
  performance while outperforming existing models across a range of
  interpretability metrics derived from the literature. Code is available at
  https://github.com/hturbe/proto-fm.
paper_authors._id: 67c42356054ae6d1c760b643
paper_authors.name: Hugues Turbé
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-02T20:15:04.391Z
paper_authors._id: 66588b6fd22637bfab498709
paper_authors.avatarUrl: /avatars/9007f0d3b078bd6193912a5359107f24.svg
paper_authors.fullname: Hugues Turbé
paper_authors.type: user
paper_authors.user: hturbe
paper_authors._id: 67c42356054ae6d1c760b644
paper_authors.name: Mina Bjelogrlic
paper_authors._id: 67c42356054ae6d1c760b645
paper_authors.name: Gianmarco Mengaldo
paper_authors._id: 67c42356054ae6d1c760b646
paper_authors.name: Christian Lovis
paper_title: Tell me why: Visual foundation models as self-explainable classifiers
paper_summary: Visual foundation models (VFMs) have become increasingly popular due to their
  state-of-the-art performance. However, interpretability remains crucial for
  critical applications. In this sense, self-explainable models (SEM) aim to
  provide interpretable classifiers that decompose predictions into a weighted
  sum of interpretable concepts. Despite their promise, recent studies have shown
  that these explanations often lack faithfulness. In this work, we combine VFMs
  with a novel prototypical architecture and specialized training objectives. By
  training only a lightweight head (approximately 1M parameters) on top of frozen
  VFMs, our approach (ProtoFM) offers an efficient and interpretable solution.
  Evaluations demonstrate that our approach achieves competitive classification
  performance while outperforming existing models across a range of
  interpretability metrics derived from the literature. Code is available at
  https://github.com/hturbe/proto-fm.
paper_ai_keywords: self-explainable models interpretable classifiers weighted sum interpretable concepts prototypical architecture specialized training objectives lightweight head frozen VFMs ProtoFM