title: Scaling laws for language encoding models in fMRI
summary: Representations from transformer-based unidirectional language models are
  known to be effective at predicting brain responses to natural language.
  However, most studies comparing language models to brains have used GPT-2 or
  similarly sized language models. Here we tested whether larger open-source
  models such as those from the OPT and LLaMA families are better at predicting
  brain responses recorded using fMRI. Mirroring scaling results from other
  contexts, we found that brain prediction performance scales log-linearly with
  model size from 125M to 30B parameter models, with ~15% increased encoding
  performance as measured by correlation with a held-out test set across 3
  subjects. Similar log-linear behavior was observed when scaling the size of the
  fMRI training set. We also characterized scaling for acoustic encoding models
  that use HuBERT, WavLM, and Whisper, and we found comparable improvements with
  model size. A noise ceiling analysis of these large, high-performance encoding
  models showed that performance is nearing the theoretical maximum for brain
  areas such as the precuneus and higher auditory cortex. These results suggest
  that increasing scale in both models and data will yield incredibly effective
  models of language processing in the brain, enabling better scientific
  understanding as well as applications such as decoding.
paper_authors._id: 646af6d51556443f24b9ef13
paper_authors.name: Richard Antonello
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-22T09:35:01.363Z
paper_authors._id: 646b354e36505117e229b773
paper_authors.avatarUrl: /avatars/3a8c51bcb969585502667faf7a7b6496.svg
paper_authors.fullname: Richard Antonello
paper_authors.type: user
paper_authors.user: RAntonello1
paper_authors._id: 646af6d51556443f24b9ef14
paper_authors.name: Aditya Vaidya
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-23T19:46:08.812Z
paper_authors._id: 6204d06212ccba127e39aaa3
paper_authors.avatarUrl: /avatars/a35c69528ae1470593d3eb1a4cdacb88.svg
paper_authors.fullname: Aditya Vaidya
paper_authors.type: user
paper_authors.user: avaidya
paper_authors._id: 646af6d51556443f24b9ef15
paper_authors.name: Alexander G. Huth
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-22T18:01:13.171Z
paper_authors._id: 6303d6633926de1f7ec2fc42
paper_authors.avatarUrl: /avatars/4ce51df91ed381fd0b1f4d8f46b1e65a.svg
paper_authors.fullname: Alex Huth
paper_authors.type: user
paper_authors.user: alexhuth
paper_title: Scaling laws for language encoding models in fMRI
paper_summary: Representations from transformer-based unidirectional language models are
  known to be effective at predicting brain responses to natural language.
  However, most studies comparing language models to brains have used GPT-2 or
  similarly sized language models. Here we tested whether larger open-source
  models such as those from the OPT and LLaMA families are better at predicting
  brain responses recorded using fMRI. Mirroring scaling results from other
  contexts, we found that brain prediction performance scales log-linearly with
  model size from 125M to 30B parameter models, with ~15% increased encoding
  performance as measured by correlation with a held-out test set across 3
  subjects. Similar log-linear behavior was observed when scaling the size of the
  fMRI training set. We also characterized scaling for acoustic encoding models
  that use HuBERT, WavLM, and Whisper, and we found comparable improvements with
  model size. A noise ceiling analysis of these large, high-performance encoding
  models showed that performance is nearing the theoretical maximum for brain
  areas such as the precuneus and higher auditory cortex. These results suggest
  that increasing scale in both models and data will yield incredibly effective
  models of language processing in the brain, enabling better scientific
  understanding as well as applications such as decoding.
paper_ai_summary: Larger transformer-based language models like OPT and LLaMA improve brain response prediction accuracy compared to smaller models, scaling log-linearly with both model size and fMRI training set size.
paper_ai_keywords: transformer-based unidirectional language models brain responses fMRI OPT LLaMA GPT-2 model size fMRI training set HuBERT WavLM Whisper acoustic encoding models noise ceiling precuneus higher auditory cortex