title: Explaining black box text modules in natural language with language
    models
summary: Large language models (LLMs) have demonstrated remarkable prediction
  performance for a growing array of tasks. However, their rapid proliferation
  and increasing opaqueness have created a growing need for interpretability.
  Here, we ask whether we can automatically obtain natural language explanations
  for black box text modules. A "text module" is any function that maps text to a
  scalar continuous value, such as a submodule within an LLM or a fitted model of
  a brain region. "Black box" indicates that we only have access to the module's
  inputs/outputs.
    We introduce Summarize and Score (SASC), a method that takes in a text module
  and returns a natural language explanation of the module's selectivity along
  with a score for how reliable the explanation is. We study SASC in 3 contexts.
  First, we evaluate SASC on synthetic modules and find that it often recovers
  ground truth explanations. Second, we use SASC to explain modules found within
  a pre-trained BERT model, enabling inspection of the model's internals.
  Finally, we show that SASC can generate explanations for the response of
  individual fMRI voxels to language stimuli, with potential applications to
  fine-grained brain mapping. All code for using SASC and reproducing results is
  made available on Github.
paper_authors._id: 646585af5cdb9ab50f874bbe
paper_authors.name: Chandan Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-13T08:33:11.745Z
paper_authors._id: 62b0bee117fa24819dd0999c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b0bee117fa24819dd0999c/eJ7ryFQ3_ZbqV29qf4KyM.jpeg
paper_authors.fullname: Chandan Singh
paper_authors.type: user
paper_authors.user: csinva
paper_authors._id: 646585af5cdb9ab50f874bbf
paper_authors.name: Aliyah R. Hsu
paper_authors._id: 646585af5cdb9ab50f874bc0
paper_authors.name: Richard Antonello
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-22T09:36:57.674Z
paper_authors._id: 646b354e36505117e229b773
paper_authors.avatarUrl: /avatars/3a8c51bcb969585502667faf7a7b6496.svg
paper_authors.fullname: Richard Antonello
paper_authors.type: user
paper_authors.user: RAntonello1
paper_authors._id: 646585af5cdb9ab50f874bc1
paper_authors.name: Shailee Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:04:16.631Z
paper_authors._id: 62b2a167e567c41e4a1433cb
paper_authors.avatarUrl: /avatars/584f58bad340ab1418f4923fd18bc0d0.svg
paper_authors.fullname: Shailee Jain
paper_authors.type: user
paper_authors.user: shailee
paper_authors._id: 646585af5cdb9ab50f874bc2
paper_authors.name: Alexander G. Huth
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-22T18:01:35.895Z
paper_authors._id: 6303d6633926de1f7ec2fc42
paper_authors.avatarUrl: /avatars/4ce51df91ed381fd0b1f4d8f46b1e65a.svg
paper_authors.fullname: Alex Huth
paper_authors.type: user
paper_authors.user: alexhuth
paper_authors._id: 646585af5cdb9ab50f874bc3
paper_authors.name: Bin Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:04:19.932Z
paper_authors._id: 64679498374fe5728d4044d5
paper_authors.avatarUrl: /avatars/78516e1c9532900cce19f81483855db8.svg
paper_authors.fullname: Bin Yu
paper_authors.type: user
paper_authors.user: binyu000
paper_authors._id: 646585af5cdb9ab50f874bc4
paper_authors.name: Jianfeng Gao
paper_title: Explaining black box text modules in natural language with language
    models
paper_summary: Large language models (LLMs) have demonstrated remarkable prediction
  performance for a growing array of tasks. However, their rapid proliferation
  and increasing opaqueness have created a growing need for interpretability.
  Here, we ask whether we can automatically obtain natural language explanations
  for black box text modules. A "text module" is any function that maps text to a
  scalar continuous value, such as a submodule within an LLM or a fitted model of
  a brain region. "Black box" indicates that we only have access to the module's
  inputs/outputs.
    We introduce Summarize and Score (SASC), a method that takes in a text module
  and returns a natural language explanation of the module's selectivity along
  with a score for how reliable the explanation is. We study SASC in 3 contexts.
  First, we evaluate SASC on synthetic modules and find that it often recovers
  ground truth explanations. Second, we use SASC to explain modules found within
  a pre-trained BERT model, enabling inspection of the model's internals.
  Finally, we show that SASC can generate explanations for the response of
  individual fMRI voxels to language stimuli, with potential applications to
  fine-grained brain mapping. All code for using SASC and reproducing results is
  made available on Github.
paper_ai_summary: SASC, a method for generating natural language explanations for black-box text modules, demonstrates reliability in synthetic, BERT model, and fMRI voxel contexts.
paper_ai_keywords: large language models synthetic modules Summarize and Score (SASC) black box text module pre-trained BERT model fMRI voxels language stimuli