title: Incorporating brain-inspired mechanisms for multimodal learning in
    artificial intelligence
summary: Multimodal learning enhances the perceptual capabilities of cognitive systems
  by integrating information from different sensory modalities. However, existing
  multimodal fusion research typically assumes static integration, not fully
  incorporating key dynamic mechanisms found in the brain. Specifically, the
  brain exhibits an inverse effectiveness phenomenon, wherein weaker unimodal
  cues yield stronger multisensory integration benefits; conversely, when
  individual modal cues are stronger, the effect of fusion is diminished. This
  mechanism enables biological systems to achieve robust cognition even with
  scarce or noisy perceptual cues. Inspired by this biological mechanism, we
  explore the relationship between multimodal output and information from
  individual modalities, proposing an inverse effectiveness driven multimodal
  fusion (IEMF) strategy. By incorporating this strategy into neural networks, we
  achieve more efficient integration with improved model performance and
  computational efficiency, demonstrating up to 50% reduction in computational
  cost across diverse fusion methods. We conduct experiments on audio-visual
  classification, continual learning, and question answering tasks to validate
  our method. Results consistently demonstrate that our method performs
  excellently in these tasks. To verify universality and generalization, we also
  conduct experiments on Artificial Neural Networks (ANN) and Spiking Neural
  Networks (SNN), with results showing good adaptability to both network types.
  Our research emphasizes the potential of incorporating biologically inspired
  mechanisms into multimodal networks and provides promising directions for the
  future development of multimodal artificial intelligence. The code is available
  at https://github.com/Brain-Cog-Lab/IEMF.
paper_authors._id: 682dbb6ed675b2cd48589a13
paper_authors.name: Xiang He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T12:26:46.945Z
paper_authors._id: 647f1e2ed26579210f5a6982
paper_authors.avatarUrl: /avatars/742f3c701de9c846f58e693aec07cfc0.svg
paper_authors.fullname: hexiang
paper_authors.type: user
paper_authors.user: xianghe
paper_authors._id: 682dbb6ed675b2cd48589a14
paper_authors.name: Dongcheng Zhao
paper_authors._id: 682dbb6ed675b2cd48589a15
paper_authors.name: Yang Li
paper_authors._id: 682dbb6ed675b2cd48589a16
paper_authors.name: Qingqun Kong
paper_authors._id: 682dbb6ed675b2cd48589a17
paper_authors.name: Xin Yang
paper_authors._id: 682dbb6ed675b2cd48589a18
paper_authors.name: Yi Zeng
paper_title: Incorporating brain-inspired mechanisms for multimodal learning in
    artificial intelligence
paper_summary: Multimodal learning enhances the perceptual capabilities of cognitive systems
  by integrating information from different sensory modalities. However, existing
  multimodal fusion research typically assumes static integration, not fully
  incorporating key dynamic mechanisms found in the brain. Specifically, the
  brain exhibits an inverse effectiveness phenomenon, wherein weaker unimodal
  cues yield stronger multisensory integration benefits; conversely, when
  individual modal cues are stronger, the effect of fusion is diminished. This
  mechanism enables biological systems to achieve robust cognition even with
  scarce or noisy perceptual cues. Inspired by this biological mechanism, we
  explore the relationship between multimodal output and information from
  individual modalities, proposing an inverse effectiveness driven multimodal
  fusion (IEMF) strategy. By incorporating this strategy into neural networks, we
  achieve more efficient integration with improved model performance and
  computational efficiency, demonstrating up to 50% reduction in computational
  cost across diverse fusion methods. We conduct experiments on audio-visual
  classification, continual learning, and question answering tasks to validate
  our method. Results consistently demonstrate that our method performs
  excellently in these tasks. To verify universality and generalization, we also
  conduct experiments on Artificial Neural Networks (ANN) and Spiking Neural
  Networks (SNN), with results showing good adaptability to both network types.
  Our research emphasizes the potential of incorporating biologically inspired
  mechanisms into multimodal networks and provides promising directions for the
  future development of multimodal artificial intelligence. The code is available
  at https://github.com/Brain-Cog-Lab/IEMF.
paper_ai_summary: Incorporating a biologically inspired inverse effectiveness mechanism into neural networks improves multimodal fusion performance and efficiency across various tasks and network types.
paper_ai_keywords: multimodal learning inverse effectiveness phenomenon multimodal fusion neural networks computational efficiency audio-visual classification continual learning question answering Artificial Neural Networks Spiking Neural Networks