title: MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
summary: Reconstructions of visual perception from brain activity have improved
  tremendously, but the practical utility of such methods has been limited. This
  is because such models are trained independently per subject where each subject
  requires dozens of hours of expensive fMRI training data to attain high-quality
  results. The present work showcases high-quality reconstructions using only 1
  hour of fMRI training data. We pretrain our model across 7 subjects and then
  fine-tune on minimal data from a new subject. Our novel functional alignment
  procedure linearly maps all brain data to a shared-subject latent space,
  followed by a shared non-linear mapping to CLIP image space. We then map from
  CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP
  latents as inputs instead of text. This approach improves out-of-subject
  generalization with limited training data and also attains state-of-the-art
  image retrieval and reconstruction metrics compared to single-subject
  approaches. MindEye2 demonstrates how accurate reconstructions of perception
  are possible from a single visit to the MRI facility. All code is available on
  GitHub.
paper_authors._id: 65f9145c94fe998c968ef1f7
paper_authors.name: Paul S. Scotti
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-03-19T04:28:15.542Z
paper_authors._id: 62b707fbdd998a8b1e424fc3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b707fbdd998a8b1e424fc3/DW_YvHw06y69jOUKz9gel.png
paper_authors.fullname: Paul Scotti
paper_authors.type: user
paper_authors.user: pscotti
paper_authors._id: 65f9145c94fe998c968ef1f8
paper_authors.name: Mihir Tripathy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T10:12:27.946Z
paper_authors._id: 62b0a2a54a5f821e911fac1f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b0a2a54a5f821e911fac1f/F32sC25VL31Ct72zdoey7.png
paper_authors.fullname: Mihir Tripathy
paper_authors.type: user
paper_authors.user: mihirneal
paper_authors._id: 65f9145c94fe998c968ef1f9
paper_authors.name: Cesar Kadir Torrico Villanueva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T10:12:36.055Z
paper_authors._id: 63195476475d4fe409acc18a
paper_authors.avatarUrl: /avatars/5731550adac1dca8156d2891c26ba832.svg
paper_authors.fullname: Cesar Kadir Torrico Villanueva
paper_authors.type: user
paper_authors.user: ckadirt
paper_authors._id: 65f9145c94fe998c968ef1fa
paper_authors.name: Reese Kneeland
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T10:12:45.105Z
paper_authors._id: 637278bb2e1ff3e884631f6a
paper_authors.avatarUrl: /avatars/2a707f0282d7c5fcdbb2729da42d7cea.svg
paper_authors.fullname: Reese Kneeland
paper_authors.type: user
paper_authors.user: reesekneeland
paper_authors._id: 65f9145c94fe998c968ef1fb
paper_authors.name: Tong Chen
paper_authors._id: 65f9145c94fe998c968ef1fc
paper_authors.name: Ashutosh Narang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T10:13:03.448Z
paper_authors._id: 6410a336a78453b8d96e7e2a
paper_authors.avatarUrl: /avatars/1d8bf1a31bb07b7986777015e3cde037.svg
paper_authors.fullname: Ashutosh Narang
paper_authors.type: user
paper_authors.user: conscioustahoe
paper_authors._id: 65f9145c94fe998c968ef1fd
paper_authors.name: Charan Santhirasegaran
paper_authors._id: 65f9145c94fe998c968ef1fe
paper_authors.name: Jonathan Xu
paper_authors._id: 65f9145c94fe998c968ef1ff
paper_authors.name: Thomas Naselaris
paper_authors._id: 65f9145c94fe998c968ef200
paper_authors.name: Kenneth A. Norman
paper_authors._id: 65f9145c94fe998c968ef201
paper_authors.name: Tanishq Mathew Abraham
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T10:14:37.447Z
paper_authors._id: 6057b823861b9d53d9c4b8df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625184855691-6057b823861b9d53d9c4b8df.jpeg
paper_authors.fullname: Tanishq Abraham
paper_authors.type: user
paper_authors.user: tmabraham
paper_title: MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
paper_summary: Reconstructions of visual perception from brain activity have improved
  tremendously, but the practical utility of such methods has been limited. This
  is because such models are trained independently per subject where each subject
  requires dozens of hours of expensive fMRI training data to attain high-quality
  results. The present work showcases high-quality reconstructions using only 1
  hour of fMRI training data. We pretrain our model across 7 subjects and then
  fine-tune on minimal data from a new subject. Our novel functional alignment
  procedure linearly maps all brain data to a shared-subject latent space,
  followed by a shared non-linear mapping to CLIP image space. We then map from
  CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP
  latents as inputs instead of text. This approach improves out-of-subject
  generalization with limited training data and also attains state-of-the-art
  image retrieval and reconstruction metrics compared to single-subject
  approaches. MindEye2 demonstrates how accurate reconstructions of perception
  are possible from a single visit to the MRI facility. All code is available on
  GitHub.
paper_ai_keywords: fMRI functional alignment latent space CLIP image space pixel space Stable Diffusion XL CLIP latents image retrieval reconstruction metrics