title: Cinematic Mindscapes: High-quality Video Reconstruction from Brain
    Activity
summary: Reconstructing human vision from brain activities has been an appealing task
  that helps to understand our cognitive process. Even though recent research has
  seen great success in reconstructing static images from non-invasive brain
  recordings, work on recovering continuous visual experiences in the form of
  videos is limited. In this work, we propose Mind-Video that learns
  spatiotemporal information from continuous fMRI data of the cerebral cortex
  progressively through masked brain modeling, multimodal contrastive learning
  with spatiotemporal attention, and co-training with an augmented Stable
  Diffusion model that incorporates network temporal inflation. We show that
  high-quality videos of arbitrary frame rates can be reconstructed with
  Mind-Video using adversarial guidance. The recovered videos were evaluated with
  various semantic and pixel-level metrics. We achieved an average accuracy of
  85% in semantic classification tasks and 0.19 in structural similarity index
  (SSIM), outperforming the previous state-of-the-art by 45%. We also show that
  our model is biologically plausible and interpretable, reflecting established
  physiological processes.
paper_authors._id: 646ac12c086023e36edca0a2
paper_authors.name: Zijiao Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-22T11:57:35.099Z
paper_authors._id: 63e1ee491e5a4f34b7a83b2e
paper_authors.avatarUrl: /avatars/4a59035bbd4011af56f6d37927d73f6f.svg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: zczczc
paper_authors._id: 646ac12c086023e36edca0a3
paper_authors.name: Jiaxin Qing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-22T11:57:37.766Z
paper_authors._id: 62df8f35a03d887642b94334
paper_authors.avatarUrl: /avatars/28f97df57fa10d1bf7c6faba4daa2afd.svg
paper_authors.fullname: Jiaxin Qing
paper_authors.type: user
paper_authors.user: jqing4749
paper_authors._id: 646ac12c086023e36edca0a4
paper_authors.name: Juan Helen Zhou
paper_title: Cinematic Mindscapes: High-quality Video Reconstruction from Brain
    Activity
paper_summary: Reconstructing human vision from brain activities has been an appealing task
  that helps to understand our cognitive process. Even though recent research has
  seen great success in reconstructing static images from non-invasive brain
  recordings, work on recovering continuous visual experiences in the form of
  videos is limited. In this work, we propose Mind-Video that learns
  spatiotemporal information from continuous fMRI data of the cerebral cortex
  progressively through masked brain modeling, multimodal contrastive learning
  with spatiotemporal attention, and co-training with an augmented Stable
  Diffusion model that incorporates network temporal inflation. We show that
  high-quality videos of arbitrary frame rates can be reconstructed with
  Mind-Video using adversarial guidance. The recovered videos were evaluated with
  various semantic and pixel-level metrics. We achieved an average accuracy of
  85% in semantic classification tasks and 0.19 in structural similarity index
  (SSIM), outperforming the previous state-of-the-art by 45%. We also show that
  our model is biologically plausible and interpretable, reflecting established
  physiological processes.
paper_ai_summary: Mind-Video reconstructs high-quality videos from continuous fMRI data using masked brain modeling, multimodal contrastive learning, and co-training with a modified Stable Diffusion model, achieving superior semantic and pixel-level performance.
paper_ai_keywords: masked brain modeling multimodal contrastive learning spatiotemporal attention co-training Stable Diffusion model network temporal inflation adversarial guidance semantic classification structural similarity index (SSIM)