Cinematic Mindscapes: High-quality Video Reconstruction from Brain
  Activity
Reconstructing human vision from brain activities has been an appealing task
that helps to understand our cognitive process. Even though recent research has
seen great success in reconstructing static images from non-invasive brain
recordings, work on recovering continuous visual experiences in the form of
videos is limited. In this work, we propose Mind-Video that learns
spatiotemporal information from continuous fMRI data of the cerebral cortex
progressively through masked brain modeling, multimodal contrastive learning
with spatiotemporal attention, and co-training with an augmented Stable
Diffusion model that incorporates network temporal inflation. We show that
high-quality videos of arbitrary frame rates can be reconstructed with
Mind-Video using adversarial guidance. The recovered videos were evaluated with
various semantic and pixel-level metrics. We achieved an average accuracy of
85% in semantic classification tasks and 0.19 in structural similarity index
(SSIM), outperforming the previous state-of-the-art by 45%. We also show that
our model is biologically plausible and interpretable, reflecting established
physiological processes.
646ac12c086023e36edca0a2
Zijiao Chen
claimed_verified
2023-05-22T11:57:35.099Z
63e1ee491e5a4f34b7a83b2e
/avatars/4a59035bbd4011af56f6d37927d73f6f.svg
Chen
user
zczczc
646ac12c086023e36edca0a3
Jiaxin Qing
claimed_verified
2023-05-22T11:57:37.766Z
62df8f35a03d887642b94334
/avatars/28f97df57fa10d1bf7c6faba4daa2afd.svg
Jiaxin Qing
user
jqing4749
646ac12c086023e36edca0a4
Juan Helen Zhou
Cinematic Mindscapes: High-quality Video Reconstruction from Brain
  Activity
Reconstructing human vision from brain activities has been an appealing task
that helps to understand our cognitive process. Even though recent research has
seen great success in reconstructing static images from non-invasive brain
recordings, work on recovering continuous visual experiences in the form of
videos is limited. In this work, we propose Mind-Video that learns
spatiotemporal information from continuous fMRI data of the cerebral cortex
progressively through masked brain modeling, multimodal contrastive learning
with spatiotemporal attention, and co-training with an augmented Stable
Diffusion model that incorporates network temporal inflation. We show that
high-quality videos of arbitrary frame rates can be reconstructed with
Mind-Video using adversarial guidance. The recovered videos were evaluated with
various semantic and pixel-level metrics. We achieved an average accuracy of
85% in semantic classification tasks and 0.19 in structural similarity index
(SSIM), outperforming the previous state-of-the-art by 45%. We also show that
our model is biologically plausible and interpretable, reflecting established
physiological processes.
Mind-Video reconstructs high-quality videos from continuous fMRI data using masked brain modeling, multimodal contrastive learning, and co-training with a modified Stable Diffusion model, achieving superior semantic and pixel-level performance.
masked brain modeling multimodal contrastive learning spatiotemporal attention co-training Stable Diffusion model network temporal inflation adversarial guidance semantic classification structural similarity index (SSIM)