Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning
  and Diffusion Priors
We present MindEye, a novel fMRI-to-image approach to retrieve and
reconstruct viewed images from brain activity. Our model comprises two parallel
submodules that are specialized for retrieval (using contrastive learning) and
reconstruction (using a diffusion prior). MindEye can map fMRI brain activity
to any high dimensional multimodal latent space, like CLIP image space,
enabling image reconstruction using generative models that accept embeddings
from this latent space. We comprehensively compare our approach with other
existing methods, using both qualitative side-by-side comparisons and
quantitative evaluations, and show that MindEye achieves state-of-the-art
performance in both reconstruction and retrieval tasks. In particular, MindEye
can retrieve the exact original image even among highly similar candidates
indicating that its brain embeddings retain fine-grained image-specific
information. This allows us to accurately retrieve images even from large-scale
databases like LAION-5B. We demonstrate through ablations that MindEye's
performance improvements over previous methods result from specialized
submodules for retrieval and reconstruction, improved training techniques, and
training models with orders of magnitude more parameters. Furthermore, we show
that MindEye can better preserve low-level image features in the
reconstructions by using img2img, with outputs from a separate autoencoder. All
code is available on GitHub.
64756bd7b68461d5cf7ef512
Paul S. Scotti
claimed_verified
2023-05-30T11:11:33.284Z
62b707fbdd998a8b1e424fc3
https://cdn-avatars.huggingface.co/v1/production/uploads/62b707fbdd998a8b1e424fc3/DW_YvHw06y69jOUKz9gel.png
Paul Scotti
user
pscotti
64756bd7b68461d5cf7ef513
Atmadeep Banerjee
64756bd7b68461d5cf7ef514
Jimmie Goode
claimed_verified
2023-05-30T14:15:17.526Z
613e5a19fdb9ea2339978745
https://cdn-avatars.huggingface.co/v1/production/uploads/613e5a19fdb9ea2339978745/Iv0UjOrXoxom8EU-EEdk3.png
Jimmie Goode
user
jimgoo
64756bd7b68461d5cf7ef515
Stepan Shabalin
admin_assigned
2023-05-30T08:58:52.757Z
613dbca582b4af22cbd7fdb9
https://cdn-avatars.huggingface.co/v1/production/uploads/1631435931286-noauth.jpeg
nev
user
nev
64756bd7b68461d5cf7ef516
Alex Nguyen
64756bd7b68461d5cf7ef517
Ethan Cohen
64756bd7b68461d5cf7ef518
Aidan J. Dempster
claimed_verified
2023-05-30T09:10:26.072Z
62b0d296476825d4e772e762
/avatars/9df351257b0d5f2ffcb3e1a48aff6aa7.svg
Aidan Dempster
user
Veldrovive
64756bd7b68461d5cf7ef519
Nathalie Verlinde
64756bd7b68461d5cf7ef51a
Elad Yundler
claimed_verified
2023-05-30T18:00:18.278Z
6303d5350907b9a115c42a5f
/avatars/eba35bf756ae1703e10cace3aa6674a5.svg
Elad Yundler
user
elad619
64756bd7b68461d5cf7ef51b
David Weisberg
64756bd7b68461d5cf7ef51c
Kenneth A. Norman
64756bd7b68461d5cf7ef51d
Tanishq Mathew Abraham
claimed_verified
2023-05-30T09:10:55.780Z
6057b823861b9d53d9c4b8df
https://cdn-avatars.huggingface.co/v1/production/uploads/1625184855691-6057b823861b9d53d9c4b8df.jpeg
Tanishq Abraham
user
tmabraham
Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning
  and Diffusion Priors
We present MindEye, a novel fMRI-to-image approach to retrieve and
reconstruct viewed images from brain activity. Our model comprises two parallel
submodules that are specialized for retrieval (using contrastive learning) and
reconstruction (using a diffusion prior). MindEye can map fMRI brain activity
to any high dimensional multimodal latent space, like CLIP image space,
enabling image reconstruction using generative models that accept embeddings
from this latent space. We comprehensively compare our approach with other
existing methods, using both qualitative side-by-side comparisons and
quantitative evaluations, and show that MindEye achieves state-of-the-art
performance in both reconstruction and retrieval tasks. In particular, MindEye
can retrieve the exact original image even among highly similar candidates
indicating that its brain embeddings retain fine-grained image-specific
information. This allows us to accurately retrieve images even from large-scale
databases like LAION-5B. We demonstrate through ablations that MindEye's
performance improvements over previous methods result from specialized
submodules for retrieval and reconstruction, improved training techniques, and
training models with orders of magnitude more parameters. Furthermore, we show
that MindEye can better preserve low-level image features in the
reconstructions by using img2img, with outputs from a separate autoencoder. All
code is available on GitHub.
MindEye, a novel fMRI-to-image approach, retrieves and reconstructs viewed images with high accuracy using specialized submodules and diffusion prior in latent space.
fMRI contrastive learning diffusion prior latent space CLIP image space generative models retrieval reconstruction LAION-5B img2img autoencoder