title: Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual
    Large Language Models
summary: The goal of this work is to enhance balanced multimodal understanding in
  audio-visual large language models (AV-LLMs) by addressing modality bias
  without requiring additional training. In current AV-LLMs, audio and video
  features are typically processed jointly in the decoder. While this strategy
  facilitates unified multimodal understanding, it may introduce modality bias,
  where the model tends to over-rely on one modality due to imbalanced training
  signals. To mitigate this, we propose Fork-Merge Decoding (FMD), a simple yet
  effective inference-time strategy that requires no additional training or
  architectural modifications. FMD first performs modality-specific reasoning by
  processing audio-only and video-only inputs through the early decoder layers (a
  fork phase), and then merges the resulting hidden states for joint reasoning in
  the remaining layers (a merge phase). This approach promotes balanced modality
  contributions and leverages complementary information across modalities. We
  evaluate our method on two representative AV-LLMs, VideoLLaMA2 and
  video-SALMONN, using three benchmark datasets. Experimental results demonstrate
  consistent performance improvements on tasks focused on audio, video, and
  combined audio-visual reasoning, demonstrating the effectiveness of
  inference-time interventions for robust multimodal understanding.
paper_authors._id: 68395a548ead63ba096bba45
paper_authors.name: Chaeyoung Jung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:46:20.597Z
paper_authors._id: 6770efb5b673f241332fc4a7
paper_authors.avatarUrl: /avatars/7aca599492233e6a51c2d6a8f52c644e.svg
paper_authors.fullname: Chaeyoung Jung
paper_authors.type: user
paper_authors.user: Chae0
paper_authors._id: 68395a548ead63ba096bba46
paper_authors.name: Youngjoon Jang
paper_authors._id: 68395a548ead63ba096bba47
paper_authors.name: Jongmin Choi
paper_authors._id: 68395a548ead63ba096bba48
paper_authors.name: Joon Son Chung
paper_title: Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual
    Large Language Models
paper_summary: The goal of this work is to enhance balanced multimodal understanding in
  audio-visual large language models (AV-LLMs) by addressing modality bias
  without requiring additional training. In current AV-LLMs, audio and video
  features are typically processed jointly in the decoder. While this strategy
  facilitates unified multimodal understanding, it may introduce modality bias,
  where the model tends to over-rely on one modality due to imbalanced training
  signals. To mitigate this, we propose Fork-Merge Decoding (FMD), a simple yet
  effective inference-time strategy that requires no additional training or
  architectural modifications. FMD first performs modality-specific reasoning by
  processing audio-only and video-only inputs through the early decoder layers (a
  fork phase), and then merges the resulting hidden states for joint reasoning in
  the remaining layers (a merge phase). This approach promotes balanced modality
  contributions and leverages complementary information across modalities. We
  evaluate our method on two representative AV-LLMs, VideoLLaMA2 and
  video-SALMONN, using three benchmark datasets. Experimental results demonstrate
  consistent performance improvements on tasks focused on audio, video, and
  combined audio-visual reasoning, demonstrating the effectiveness of
  inference-time interventions for robust multimodal understanding.
paper_ai_summary: The Fork-Merge Decoding strategy improves balanced multimodal understanding in audio-visual large language models by separating and then combining modality-specific reasoning.
paper_ai_keywords: fork-merge decoding AU-LLMs modality bias audio-visual large language models VideoLLaMA2 video-SALMONN benchmark datasets audio-visual reasoning