Meta-Learning an In-Context Transformer Model of Human Higher Visual
  Cortex
Understanding functional representations within higher visual cortex is a
fundamental question in computational neuroscience. While artificial neural
networks pretrained on large-scale datasets exhibit striking representational
alignment with human neural responses, learning image-computable models of
visual cortex relies on individual-level, large-scale fMRI datasets. The
necessity for expensive, time-intensive, and often impractical data acquisition
limits the generalizability of encoders to new subjects and stimuli. BraInCoRL
uses in-context learning to predict voxelwise neural responses from few-shot
examples without any additional finetuning for novel subjects and stimuli. We
leverage a transformer architecture that can flexibly condition on a variable
number of in-context image stimuli, learning an inductive bias over multiple
subjects. During training, we explicitly optimize the model for in-context
learning. By jointly conditioning on image features and voxel activations, our
model learns to directly generate better performing voxelwise models of higher
visual cortex. We demonstrate that BraInCoRL consistently outperforms existing
voxelwise encoder designs in a low-data regime when evaluated on entirely novel
images, while also exhibiting strong test-time scaling behavior. The model also
generalizes to an entirely new visual fMRI dataset, which uses different
subjects and fMRI data acquisition parameters. Further, BraInCoRL facilitates
better interpretability of neural signals in higher visual cortex by attending
to semantically relevant stimuli. Finally, we show that our framework enables
interpretable mappings from natural language queries to voxel selectivity.
68380974717461677df17514
Muquan Yu
68380974717461677df17515
Mu Nan
68380974717461677df17516
Hossein Adeli
claimed_verified
2025-07-02T07:41:32.719Z
650f87d0463e7e33e97c2ec6
/avatars/f3515a62cdd36d4496a0abe6ab1dfbc2.svg
Hossein
user
HosseinAdeli
68380974717461677df17517
Jacob S. Prince
68380974717461677df17518
John A. Pyles
68380974717461677df17519
Leila Wehbe
68380974717461677df1751a
Margaret M. Henderson
68380974717461677df1751b
Michael J. Tarr
68380974717461677df1751c
Andrew F. Luo
extracted_confirmed
2025-05-29T07:16:44.495Z
64b6ce23dbbd1f2cdb624d56
/avatars/022738ce8f76fa9545b3e363d7264b53.svg
Andrew Luo
user
aluo-x
Meta-Learning an In-Context Transformer Model of Human Higher Visual
  Cortex
Understanding functional representations within higher visual cortex is a
fundamental question in computational neuroscience. While artificial neural
networks pretrained on large-scale datasets exhibit striking representational
alignment with human neural responses, learning image-computable models of
visual cortex relies on individual-level, large-scale fMRI datasets. The
necessity for expensive, time-intensive, and often impractical data acquisition
limits the generalizability of encoders to new subjects and stimuli. BraInCoRL
uses in-context learning to predict voxelwise neural responses from few-shot
examples without any additional finetuning for novel subjects and stimuli. We
leverage a transformer architecture that can flexibly condition on a variable
number of in-context image stimuli, learning an inductive bias over multiple
subjects. During training, we explicitly optimize the model for in-context
learning. By jointly conditioning on image features and voxel activations, our
model learns to directly generate better performing voxelwise models of higher
visual cortex. We demonstrate that BraInCoRL consistently outperforms existing
voxelwise encoder designs in a low-data regime when evaluated on entirely novel
images, while also exhibiting strong test-time scaling behavior. The model also
generalizes to an entirely new visual fMRI dataset, which uses different
subjects and fMRI data acquisition parameters. Further, BraInCoRL facilitates
better interpretability of neural signals in higher visual cortex by attending
to semantically relevant stimuli. Finally, we show that our framework enables
interpretable mappings from natural language queries to voxel selectivity.
BraInCoRL employs a transformer-based in-context learning approach to model higher visual cortex neural responses with few-shot examples, demonstrating superior performance and generalizability across new subjects, stimuli, and datasets.
functional representations higher visual cortex artificial neural networks fMRI datasets in-context learning transformer architecture inductive bias voxelwise neural responses image features voxel activations test-time scaling interpretability natural language queries voxel selectivity