Incorporating brain-inspired mechanisms for multimodal learning in
  artificial intelligence
Multimodal learning enhances the perceptual capabilities of cognitive systems
by integrating information from different sensory modalities. However, existing
multimodal fusion research typically assumes static integration, not fully
incorporating key dynamic mechanisms found in the brain. Specifically, the
brain exhibits an inverse effectiveness phenomenon, wherein weaker unimodal
cues yield stronger multisensory integration benefits; conversely, when
individual modal cues are stronger, the effect of fusion is diminished. This
mechanism enables biological systems to achieve robust cognition even with
scarce or noisy perceptual cues. Inspired by this biological mechanism, we
explore the relationship between multimodal output and information from
individual modalities, proposing an inverse effectiveness driven multimodal
fusion (IEMF) strategy. By incorporating this strategy into neural networks, we
achieve more efficient integration with improved model performance and
computational efficiency, demonstrating up to 50% reduction in computational
cost across diverse fusion methods. We conduct experiments on audio-visual
classification, continual learning, and question answering tasks to validate
our method. Results consistently demonstrate that our method performs
excellently in these tasks. To verify universality and generalization, we also
conduct experiments on Artificial Neural Networks (ANN) and Spiking Neural
Networks (SNN), with results showing good adaptability to both network types.
Our research emphasizes the potential of incorporating biologically inspired
mechanisms into multimodal networks and provides promising directions for the
future development of multimodal artificial intelligence. The code is available
at https://github.com/Brain-Cog-Lab/IEMF.
682dbb6ed675b2cd48589a13
Xiang He
claimed_verified
2025-05-21T12:26:46.945Z
647f1e2ed26579210f5a6982
/avatars/742f3c701de9c846f58e693aec07cfc0.svg
hexiang
user
xianghe
682dbb6ed675b2cd48589a14
Dongcheng Zhao
682dbb6ed675b2cd48589a15
Yang Li
682dbb6ed675b2cd48589a16
Qingqun Kong
682dbb6ed675b2cd48589a17
Xin Yang
682dbb6ed675b2cd48589a18
Yi Zeng
Incorporating brain-inspired mechanisms for multimodal learning in
  artificial intelligence
Multimodal learning enhances the perceptual capabilities of cognitive systems
by integrating information from different sensory modalities. However, existing
multimodal fusion research typically assumes static integration, not fully
incorporating key dynamic mechanisms found in the brain. Specifically, the
brain exhibits an inverse effectiveness phenomenon, wherein weaker unimodal
cues yield stronger multisensory integration benefits; conversely, when
individual modal cues are stronger, the effect of fusion is diminished. This
mechanism enables biological systems to achieve robust cognition even with
scarce or noisy perceptual cues. Inspired by this biological mechanism, we
explore the relationship between multimodal output and information from
individual modalities, proposing an inverse effectiveness driven multimodal
fusion (IEMF) strategy. By incorporating this strategy into neural networks, we
achieve more efficient integration with improved model performance and
computational efficiency, demonstrating up to 50% reduction in computational
cost across diverse fusion methods. We conduct experiments on audio-visual
classification, continual learning, and question answering tasks to validate
our method. Results consistently demonstrate that our method performs
excellently in these tasks. To verify universality and generalization, we also
conduct experiments on Artificial Neural Networks (ANN) and Spiking Neural
Networks (SNN), with results showing good adaptability to both network types.
Our research emphasizes the potential of incorporating biologically inspired
mechanisms into multimodal networks and provides promising directions for the
future development of multimodal artificial intelligence. The code is available
at https://github.com/Brain-Cog-Lab/IEMF.
Incorporating a biologically inspired inverse effectiveness mechanism into neural networks improves multimodal fusion performance and efficiency across various tasks and network types.
multimodal learning inverse effectiveness phenomenon multimodal fusion neural networks computational efficiency audio-visual classification continual learning question answering Artificial Neural Networks Spiking Neural Networks