title: See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language
    Balance to Mitigate Dominant Modality Bias
summary: Vision-language (VL) models have demonstrated strong performance across
  various tasks. However, these models often rely on a specific modality for
  predictions, leading to "dominant modality bias.'' This bias significantly
  hurts performance, especially when one modality is impaired. In this study, we
  analyze model behavior under dominant modality bias and theoretically show that
  unaligned gradients or differences in gradient magnitudes prevent balanced
  convergence of the loss. Based on these findings, we propose a novel framework,
  BalGrad to mitigate dominant modality bias. Our approach includes
  inter-modality gradient reweighting, adjusting the gradient of KL divergence
  based on each modality's contribution, and inter-task gradient projection to
  align task directions in a non-conflicting manner. Experiments on UPMC
  Food-101, Hateful Memes, and MM-IMDb datasets confirm that BalGrad effectively
  alleviates over-reliance on specific modalities when making predictions.
paper_authors._id: 67dcf6456b575dc3179e05a2
paper_authors.name: JuneHyoung Kwon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:08:35.264Z
paper_authors._id: 6756f15abba07064c254d8d3
paper_authors.avatarUrl: /avatars/0b7067c2475d5cb1950579cbb98b54f3.svg
paper_authors.fullname: June Hyoung Kwon
paper_authors.type: user
paper_authors.user: herbwood27
paper_authors._id: 67dcf6456b575dc3179e05a3
paper_authors.name: MiHyeon Kim
paper_authors._id: 67dcf6456b575dc3179e05a4
paper_authors.name: Eunju Lee
paper_authors._id: 67dcf6456b575dc3179e05a5
paper_authors.name: Juhwan Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:08:21.660Z
paper_authors._id: 65646b22ac9d3c2bd7b14788
paper_authors.avatarUrl: /avatars/0bf19dcfa568a694361fb3a63b999997.svg
paper_authors.fullname: Juhwan Choi
paper_authors.type: user
paper_authors.user: c-juhwan
paper_authors._id: 67dcf6456b575dc3179e05a6
paper_authors.name: YoungBin Kim
paper_title: See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language
    Balance to Mitigate Dominant Modality Bias
paper_summary: Vision-language (VL) models have demonstrated strong performance across
  various tasks. However, these models often rely on a specific modality for
  predictions, leading to "dominant modality bias.'' This bias significantly
  hurts performance, especially when one modality is impaired. In this study, we
  analyze model behavior under dominant modality bias and theoretically show that
  unaligned gradients or differences in gradient magnitudes prevent balanced
  convergence of the loss. Based on these findings, we propose a novel framework,
  BalGrad to mitigate dominant modality bias. Our approach includes
  inter-modality gradient reweighting, adjusting the gradient of KL divergence
  based on each modality's contribution, and inter-task gradient projection to
  align task directions in a non-conflicting manner. Experiments on UPMC
  Food-101, Hateful Memes, and MM-IMDb datasets confirm that BalGrad effectively
  alleviates over-reliance on specific modalities when making predictions.
paper_ai_summary: A framework called BalGrad is introduced to mitigate dominant modality bias in vision-language models by reweighting and aligning gradients from different modalities.
paper_ai_keywords: modality bias unaligned gradients gradient magnitudes inter-modality gradient reweighting KL divergence inter-task gradient projection UPMC Food-101 Hateful Memes MM-IMDb