title: CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic
    Audiovisual Narrative Processing
summary: In this paper, we introduce CineBrain, the first large-scale dataset
  featuring simultaneous EEG and fMRI recordings during dynamic audiovisual
  stimulation. Recognizing the complementary strengths of EEG's high temporal
  resolution and fMRI's deep-brain spatial coverage, CineBrain provides
  approximately six hours of narrative-driven content from the popular television
  series The Big Bang Theory for each of six participants. Building upon this
  unique dataset, we propose CineSync, an innovative multimodal decoding
  framework integrates a Multi-Modal Fusion Encoder with a diffusion-based Neural
  Latent Decoder. Our approach effectively fuses EEG and fMRI signals,
  significantly improving the reconstruction quality of complex audiovisual
  stimuli. To facilitate rigorous evaluation, we introduce Cine-Benchmark, a
  comprehensive evaluation protocol that assesses reconstructions across semantic
  and perceptual dimensions. Experimental results demonstrate that CineSync
  achieves state-of-the-art video reconstruction performance and highlight our
  initial success in combining fMRI and EEG for reconstructing both video and
  audio stimuli. Project Page: https://jianxgao.github.io/CineBrain.
paper_authors._id: 67d1382885d9baaf658a4b55
paper_authors.name: Jianxiong Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T14:25:40.647Z
paper_authors._id: 643815c4961bb61e463c5896
paper_authors.avatarUrl: /avatars/3b44592472f16c56105bff8c314d9939.svg
paper_authors.fullname: Jianxiong Gao
paper_authors.type: user
paper_authors.user: Jianxiong
paper_authors._id: 67d1382885d9baaf658a4b56
paper_authors.name: Yichang Liu
paper_authors._id: 67d1382885d9baaf658a4b57
paper_authors.name: Baofeng Yang
paper_authors._id: 67d1382885d9baaf658a4b58
paper_authors.name: Jianfeng Feng
paper_authors._id: 67d1382885d9baaf658a4b59
paper_authors.name: Yanwei Fu
paper_title: CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic
    Audiovisual Narrative Processing
paper_summary: In this paper, we introduce CineBrain, the first large-scale dataset
  featuring simultaneous EEG and fMRI recordings during dynamic audiovisual
  stimulation. Recognizing the complementary strengths of EEG's high temporal
  resolution and fMRI's deep-brain spatial coverage, CineBrain provides
  approximately six hours of narrative-driven content from the popular television
  series The Big Bang Theory for each of six participants. Building upon this
  unique dataset, we propose CineSync, an innovative multimodal decoding
  framework integrates a Multi-Modal Fusion Encoder with a diffusion-based Neural
  Latent Decoder. Our approach effectively fuses EEG and fMRI signals,
  significantly improving the reconstruction quality of complex audiovisual
  stimuli. To facilitate rigorous evaluation, we introduce Cine-Benchmark, a
  comprehensive evaluation protocol that assesses reconstructions across semantic
  and perceptual dimensions. Experimental results demonstrate that CineSync
  achieves state-of-the-art video reconstruction performance and highlight our
  initial success in combining fMRI and EEG for reconstructing both video and
  audio stimuli. Project Page: https://jianxgao.github.io/CineBrain.
paper_ai_summary: CineSync, a multimodal decoding framework using reinforcement learning with a diffusion-based neural latent decoder, achieves state-of-the-art performance in reconstructing complex audiovisual stimuli from EEG and fMRI data.
paper_ai_keywords: EEG fMRI Multi-Modal Fusion Encoder diffusion-based Neural Latent Decoder CineSync CineBrain Cine-Benchmark