title: fMRI-3D: A Comprehensive Dataset for Enhancing fMRI-based 3D
    Reconstruction
summary: Reconstructing 3D visuals from functional Magnetic Resonance Imaging (fMRI)
  data, introduced as Recon3DMind in our conference work, is of significant
  interest to both cognitive neuroscience and computer vision. To advance this
  task, we present the fMRI-3D dataset, which includes data from 15 participants
  and showcases a total of 4768 3D objects. The dataset comprises two components:
  fMRI-Shape, previously introduced and accessible at
  https://huggingface.co/datasets/Fudan-fMRI/fMRI-Shape, and fMRI-Objaverse,
  proposed in this paper and available at
  https://huggingface.co/datasets/Fudan-fMRI/fMRI-Objaverse. fMRI-Objaverse
  includes data from 5 subjects, 4 of whom are also part of the Core set in
  fMRI-Shape, with each subject viewing 3142 3D objects across 117 categories,
  all accompanied by text captions. This significantly enhances the diversity and
  potential applications of the dataset. Additionally, we propose MinD-3D, a
  novel framework designed to decode 3D visual information from fMRI signals. The
  framework first extracts and aggregates features from fMRI data using a
  neuro-fusion encoder, then employs a feature-bridge diffusion model to generate
  visual features, and finally reconstructs the 3D object using a generative
  transformer decoder. We establish new benchmarks by designing metrics at both
  semantic and structural levels to evaluate model performance. Furthermore, we
  assess our model's effectiveness in an Out-of-Distribution setting and analyze
  the attribution of the extracted features and the visual ROIs in fMRI signals.
  Our experiments demonstrate that MinD-3D not only reconstructs 3D objects with
  high semantic and spatial accuracy but also deepens our understanding of how
  human brain processes 3D visual information. Project page at:
  https://jianxgao.github.io/MinD-3D.
paper_authors._id: 66ea5152c764a0ee6e507f1a
paper_authors.name: Jianxiong Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:34.398Z
paper_authors._id: 643815c4961bb61e463c5896
paper_authors.avatarUrl: /avatars/3b44592472f16c56105bff8c314d9939.svg
paper_authors.fullname: Jianxiong Gao
paper_authors.type: user
paper_authors.user: Jianxiong
paper_authors._id: 66ea5152c764a0ee6e507f1b
paper_authors.name: Yuqian Fu
paper_authors._id: 66ea5152c764a0ee6e507f1c
paper_authors.name: Yun Wang
paper_authors._id: 66ea5152c764a0ee6e507f1d
paper_authors.name: Xuelin Qian
paper_authors._id: 66ea5152c764a0ee6e507f1e
paper_authors.name: Jianfeng Feng
paper_authors._id: 66ea5152c764a0ee6e507f1f
paper_authors.name: Yanwei Fu
paper_title: fMRI-3D: A Comprehensive Dataset for Enhancing fMRI-based 3D
    Reconstruction
paper_summary: Reconstructing 3D visuals from functional Magnetic Resonance Imaging (fMRI)
  data, introduced as Recon3DMind in our conference work, is of significant
  interest to both cognitive neuroscience and computer vision. To advance this
  task, we present the fMRI-3D dataset, which includes data from 15 participants
  and showcases a total of 4768 3D objects. The dataset comprises two components:
  fMRI-Shape, previously introduced and accessible at
  https://huggingface.co/datasets/Fudan-fMRI/fMRI-Shape, and fMRI-Objaverse,
  proposed in this paper and available at
  https://huggingface.co/datasets/Fudan-fMRI/fMRI-Objaverse. fMRI-Objaverse
  includes data from 5 subjects, 4 of whom are also part of the Core set in
  fMRI-Shape, with each subject viewing 3142 3D objects across 117 categories,
  all accompanied by text captions. This significantly enhances the diversity and
  potential applications of the dataset. Additionally, we propose MinD-3D, a
  novel framework designed to decode 3D visual information from fMRI signals. The
  framework first extracts and aggregates features from fMRI data using a
  neuro-fusion encoder, then employs a feature-bridge diffusion model to generate
  visual features, and finally reconstructs the 3D object using a generative
  transformer decoder. We establish new benchmarks by designing metrics at both
  semantic and structural levels to evaluate model performance. Furthermore, we
  assess our model's effectiveness in an Out-of-Distribution setting and analyze
  the attribution of the extracted features and the visual ROIs in fMRI signals.
  Our experiments demonstrate that MinD-3D not only reconstructs 3D objects with
  high semantic and spatial accuracy but also deepens our understanding of how
  human brain processes 3D visual information. Project page at:
  https://jianxgao.github.io/MinD-3D.
paper_ai_keywords: neuro-fusion encoder feature-bridge diffusion model generative transformer decoder semantic levels structural levels MinD-3D