title: Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning
    and Diffusion Priors
summary: We present MindEye, a novel fMRI-to-image approach to retrieve and
  reconstruct viewed images from brain activity. Our model comprises two parallel
  submodules that are specialized for retrieval (using contrastive learning) and
  reconstruction (using a diffusion prior). MindEye can map fMRI brain activity
  to any high dimensional multimodal latent space, like CLIP image space,
  enabling image reconstruction using generative models that accept embeddings
  from this latent space. We comprehensively compare our approach with other
  existing methods, using both qualitative side-by-side comparisons and
  quantitative evaluations, and show that MindEye achieves state-of-the-art
  performance in both reconstruction and retrieval tasks. In particular, MindEye
  can retrieve the exact original image even among highly similar candidates
  indicating that its brain embeddings retain fine-grained image-specific
  information. This allows us to accurately retrieve images even from large-scale
  databases like LAION-5B. We demonstrate through ablations that MindEye's
  performance improvements over previous methods result from specialized
  submodules for retrieval and reconstruction, improved training techniques, and
  training models with orders of magnitude more parameters. Furthermore, we show
  that MindEye can better preserve low-level image features in the
  reconstructions by using img2img, with outputs from a separate autoencoder. All
  code is available on GitHub.
paper_authors._id: 64756bd7b68461d5cf7ef512
paper_authors.name: Paul S. Scotti
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T11:11:33.284Z
paper_authors._id: 62b707fbdd998a8b1e424fc3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b707fbdd998a8b1e424fc3/DW_YvHw06y69jOUKz9gel.png
paper_authors.fullname: Paul Scotti
paper_authors.type: user
paper_authors.user: pscotti
paper_authors._id: 64756bd7b68461d5cf7ef513
paper_authors.name: Atmadeep Banerjee
paper_authors._id: 64756bd7b68461d5cf7ef514
paper_authors.name: Jimmie Goode
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T14:15:17.526Z
paper_authors._id: 613e5a19fdb9ea2339978745
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/613e5a19fdb9ea2339978745/Iv0UjOrXoxom8EU-EEdk3.png
paper_authors.fullname: Jimmie Goode
paper_authors.type: user
paper_authors.user: jimgoo
paper_authors._id: 64756bd7b68461d5cf7ef515
paper_authors.name: Stepan Shabalin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-30T08:58:52.757Z
paper_authors._id: 613dbca582b4af22cbd7fdb9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631435931286-noauth.jpeg
paper_authors.fullname: nev
paper_authors.type: user
paper_authors.user: nev
paper_authors._id: 64756bd7b68461d5cf7ef516
paper_authors.name: Alex Nguyen
paper_authors._id: 64756bd7b68461d5cf7ef517
paper_authors.name: Ethan Cohen
paper_authors._id: 64756bd7b68461d5cf7ef518
paper_authors.name: Aidan J. Dempster
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T09:10:26.072Z
paper_authors._id: 62b0d296476825d4e772e762
paper_authors.avatarUrl: /avatars/9df351257b0d5f2ffcb3e1a48aff6aa7.svg
paper_authors.fullname: Aidan Dempster
paper_authors.type: user
paper_authors.user: Veldrovive
paper_authors._id: 64756bd7b68461d5cf7ef519
paper_authors.name: Nathalie Verlinde
paper_authors._id: 64756bd7b68461d5cf7ef51a
paper_authors.name: Elad Yundler
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T18:00:18.278Z
paper_authors._id: 6303d5350907b9a115c42a5f
paper_authors.avatarUrl: /avatars/eba35bf756ae1703e10cace3aa6674a5.svg
paper_authors.fullname: Elad Yundler
paper_authors.type: user
paper_authors.user: elad619
paper_authors._id: 64756bd7b68461d5cf7ef51b
paper_authors.name: David Weisberg
paper_authors._id: 64756bd7b68461d5cf7ef51c
paper_authors.name: Kenneth A. Norman
paper_authors._id: 64756bd7b68461d5cf7ef51d
paper_authors.name: Tanishq Mathew Abraham
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T09:10:55.780Z
paper_authors._id: 6057b823861b9d53d9c4b8df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625184855691-6057b823861b9d53d9c4b8df.jpeg
paper_authors.fullname: Tanishq Abraham
paper_authors.type: user
paper_authors.user: tmabraham
paper_title: Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning
    and Diffusion Priors
paper_summary: We present MindEye, a novel fMRI-to-image approach to retrieve and
  reconstruct viewed images from brain activity. Our model comprises two parallel
  submodules that are specialized for retrieval (using contrastive learning) and
  reconstruction (using a diffusion prior). MindEye can map fMRI brain activity
  to any high dimensional multimodal latent space, like CLIP image space,
  enabling image reconstruction using generative models that accept embeddings
  from this latent space. We comprehensively compare our approach with other
  existing methods, using both qualitative side-by-side comparisons and
  quantitative evaluations, and show that MindEye achieves state-of-the-art
  performance in both reconstruction and retrieval tasks. In particular, MindEye
  can retrieve the exact original image even among highly similar candidates
  indicating that its brain embeddings retain fine-grained image-specific
  information. This allows us to accurately retrieve images even from large-scale
  databases like LAION-5B. We demonstrate through ablations that MindEye's
  performance improvements over previous methods result from specialized
  submodules for retrieval and reconstruction, improved training techniques, and
  training models with orders of magnitude more parameters. Furthermore, we show
  that MindEye can better preserve low-level image features in the
  reconstructions by using img2img, with outputs from a separate autoencoder. All
  code is available on GitHub.
paper_ai_summary: MindEye, a novel fMRI-to-image approach, retrieves and reconstructs viewed images with high accuracy using specialized submodules and diffusion prior in latent space.
paper_ai_keywords: fMRI contrastive learning diffusion prior latent space CLIP image space generative models retrieval reconstruction LAION-5B img2img autoencoder