title: Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal
    Representations
summary: Prior research on out-of-distribution detection (OoDD) has primarily focused
  on single-modality models. Recently, with the advent of large-scale pretrained
  vision-language models such as CLIP, OoDD methods utilizing such multi-modal
  representations through zero-shot and prompt learning strategies have emerged.
  However, these methods typically involve either freezing the pretrained weights
  or only partially tuning them, which can be suboptimal for downstream datasets.
  In this paper, we highlight that multi-modal fine-tuning (MMFT) can achieve
  notable OoDD performance. Despite some recent works demonstrating the impact of
  fine-tuning methods for OoDD, there remains significant potential for
  performance improvement. We investigate the limitation of na\"ive fine-tuning
  methods, examining why they fail to fully leverage the pretrained knowledge.
  Our empirical analysis suggests that this issue could stem from the modality
  gap within in-distribution (ID) embeddings. To address this, we propose a
  training objective that enhances cross-modal alignment by regularizing the
  distances between image and text embeddings of ID data. This adjustment helps
  in better utilizing pretrained textual information by aligning similar
  semantics from different modalities (i.e., text and image) more closely in the
  hyperspherical representation space. We theoretically demonstrate that the
  proposed regularization corresponds to the maximum likelihood estimation of an
  energy-based model on a hypersphere. Utilizing ImageNet-1k OoD benchmark
  datasets, we show that our method, combined with post-hoc OoDD approaches
  leveraging pretrained knowledge (e.g., NegLabel), significantly outperforms
  existing methods, achieving state-of-the-art OoDD performance and leading ID
  accuracy.
paper_authors._id: 67ede492bdd88c72dc99fbd7
paper_authors.name: Jeonghyeon Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T07:52:31.201Z
paper_authors._id: 654b4c9cfabd2cc66874806c
paper_authors.avatarUrl: /avatars/c48b423d3657ee27eed0d3d83738ec2e.svg
paper_authors.fullname: jeonghyeon kim
paper_authors.type: user
paper_authors.user: mawjdgus
paper_authors._id: 67ede492bdd88c72dc99fbd8
paper_authors.name: Sangheum Hwang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T14:07:05.202Z
paper_authors._id: 64e6ff6b09375662071ae8c8
paper_authors.avatarUrl: /avatars/4e1086476a29733eb4680eb8771d2f68.svg
paper_authors.fullname: Sangheum Hwang
paper_authors.type: user
paper_authors.user: beopst
paper_title: Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal
    Representations
paper_summary: Prior research on out-of-distribution detection (OoDD) has primarily focused
  on single-modality models. Recently, with the advent of large-scale pretrained
  vision-language models such as CLIP, OoDD methods utilizing such multi-modal
  representations through zero-shot and prompt learning strategies have emerged.
  However, these methods typically involve either freezing the pretrained weights
  or only partially tuning them, which can be suboptimal for downstream datasets.
  In this paper, we highlight that multi-modal fine-tuning (MMFT) can achieve
  notable OoDD performance. Despite some recent works demonstrating the impact of
  fine-tuning methods for OoDD, there remains significant potential for
  performance improvement. We investigate the limitation of na\"ive fine-tuning
  methods, examining why they fail to fully leverage the pretrained knowledge.
  Our empirical analysis suggests that this issue could stem from the modality
  gap within in-distribution (ID) embeddings. To address this, we propose a
  training objective that enhances cross-modal alignment by regularizing the
  distances between image and text embeddings of ID data. This adjustment helps
  in better utilizing pretrained textual information by aligning similar
  semantics from different modalities (i.e., text and image) more closely in the
  hyperspherical representation space. We theoretically demonstrate that the
  proposed regularization corresponds to the maximum likelihood estimation of an
  energy-based model on a hypersphere. Utilizing ImageNet-1k OoD benchmark
  datasets, we show that our method, combined with post-hoc OoDD approaches
  leveraging pretrained knowledge (e.g., NegLabel), significantly outperforms
  existing methods, achieving state-of-the-art OoDD performance and leading ID
  accuracy.
paper_ai_summary: A novel training objective that enhances cross-modal alignment through regularization of embedding distances improves out-of-distribution detection and in-distribution accuracy using multi-modal fine-tuning.
paper_ai_keywords: out-of-distribution detection OoDD single-modality models vision-language models CLIP zero-shot learning prompt learning multi-modal fine-tuning MMFT na\"ive fine-tuning modality gap in-distribution embeddings cross-modal alignment energy-based model hyperspherical representation space ImageNet-1k NegLabel