title: Studying Large Language Model Generalization with Influence Functions
summary: When trying to gain better visibility into a machine learning model in order
  to understand and mitigate the associated risks, a potentially valuable source
  of evidence is: which training examples most contribute to a given behavior?
  Influence functions aim to answer a counterfactual: how would the model's
  parameters (and hence its outputs) change if a given sequence were added to the
  training set? While influence functions have produced insights for small
  models, they are difficult to scale to large language models (LLMs) due to the
  difficulty of computing an inverse-Hessian-vector product (IHVP). We use the
  Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC)
  approximation to scale influence functions up to LLMs with up to 52 billion
  parameters. In our experiments, EK-FAC achieves similar accuracy to traditional
  influence function estimators despite the IHVP computation being orders of
  magnitude faster. We investigate two algorithmic techniques to reduce the cost
  of computing gradients of candidate training sequences: TF-IDF filtering and
  query batching. We use influence functions to investigate the generalization
  patterns of LLMs, including the sparsity of the influence patterns, increasing
  abstraction with scale, math and programming abilities, cross-lingual
  generalization, and role-playing behavior. Despite many apparently
  sophisticated forms of generalization, we identify a surprising limitation:
  influences decay to near-zero when the order of key phrases is flipped.
  Overall, influence functions give us a powerful new tool for studying the
  generalization properties of LLMs.
paper_authors._id: 64d1ac192f1f9578a0661d59
paper_authors.name: Roger Grosse
paper_authors._id: 64d1ac192f1f9578a0661d5a
paper_authors.name: Juhan Bae
paper_authors._id: 64d1ac192f1f9578a0661d5b
paper_authors.name: Cem Anil
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T15:03:38.323Z
paper_authors._id: 62f6b27d05ca68c0e0008549
paper_authors.avatarUrl: /avatars/14d0075aa1b578cd7ee5f9e68d12e2f0.svg
paper_authors.fullname: Cem Anil
paper_authors.type: user
paper_authors.user: anilcem
paper_authors._id: 64d1ac192f1f9578a0661d5c
paper_authors.name: Nelson Elhage
paper_authors._id: 64d1ac192f1f9578a0661d5d
paper_authors.name: Alex Tamkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T15:05:31.550Z
paper_authors._id: 5fc18c42ea82dd667bb0ffaf
paper_authors.avatarUrl: /avatars/ae98f82442b560de247d2354cc364398.svg
paper_authors.fullname: Alex Tamkin
paper_authors.type: user
paper_authors.user: alextamkin
paper_authors._id: 64d1ac192f1f9578a0661d5e
paper_authors.name: Amirhossein Tajdini
paper_authors._id: 64d1ac192f1f9578a0661d5f
paper_authors.name: Benoit Steiner
paper_authors._id: 64d1ac192f1f9578a0661d60
paper_authors.name: Dustin Li
paper_authors._id: 64d1ac192f1f9578a0661d61
paper_authors.name: Esin Durmus
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T15:06:01.415Z
paper_authors._id: 627dd7dbba19b19eff6aaa46
paper_authors.avatarUrl: /avatars/6e43025ccb96e48339d1db477da043b0.svg
paper_authors.fullname: Esin Durmus
paper_authors.type: user
paper_authors.user: esdurmus
paper_authors._id: 64d1ac192f1f9578a0661d62
paper_authors.name: Ethan Perez
paper_authors._id: 64d1ac192f1f9578a0661d63
paper_authors.name: Evan Hubinger
paper_authors._id: 64d1ac192f1f9578a0661d64
paper_authors.name: Kamilė Lukošiūtė
paper_authors._id: 64d1ac192f1f9578a0661d65
paper_authors.name: Karina Nguyen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T15:06:37.243Z
paper_authors._id: 62ef4469cbe9a53fcd5a7f03
paper_authors.avatarUrl: /avatars/13352152a67ef304ba444cbc310be5dd.svg
paper_authors.fullname: Karina Nguyen
paper_authors.type: user
paper_authors.user: karinanguyen
paper_authors._id: 64d1ac192f1f9578a0661d66
paper_authors.name: Nicholas Joseph
paper_authors._id: 64d1ac192f1f9578a0661d67
paper_authors.name: Sam McCandlish
paper_authors._id: 64d1ac192f1f9578a0661d68
paper_authors.name: Jared Kaplan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T15:07:02.324Z
paper_authors._id: 633b051c7af633cbcd01b0ff
paper_authors.avatarUrl: /avatars/e2f315a8d6241ae1a783ba2957a211a5.svg
paper_authors.fullname: Jared Kaplan
paper_authors.type: user
paper_authors.user: FrizzleFried
paper_authors._id: 64d1ac192f1f9578a0661d69
paper_authors.name: Samuel R. Bowman
paper_title: Studying Large Language Model Generalization with Influence Functions
paper_summary: When trying to gain better visibility into a machine learning model in order
  to understand and mitigate the associated risks, a potentially valuable source
  of evidence is: which training examples most contribute to a given behavior?
  Influence functions aim to answer a counterfactual: how would the model's
  parameters (and hence its outputs) change if a given sequence were added to the
  training set? While influence functions have produced insights for small
  models, they are difficult to scale to large language models (LLMs) due to the
  difficulty of computing an inverse-Hessian-vector product (IHVP). We use the
  Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC)
  approximation to scale influence functions up to LLMs with up to 52 billion
  parameters. In our experiments, EK-FAC achieves similar accuracy to traditional
  influence function estimators despite the IHVP computation being orders of
  magnitude faster. We investigate two algorithmic techniques to reduce the cost
  of computing gradients of candidate training sequences: TF-IDF filtering and
  query batching. We use influence functions to investigate the generalization
  patterns of LLMs, including the sparsity of the influence patterns, increasing
  abstraction with scale, math and programming abilities, cross-lingual
  generalization, and role-playing behavior. Despite many apparently
  sophisticated forms of generalization, we identify a surprising limitation:
  influences decay to near-zero when the order of key phrases is flipped.
  Overall, influence functions give us a powerful new tool for studying the
  generalization properties of LLMs.
paper_ai_keywords: influence functions counterfactual inverse-Hessian-vector product (IHVP) Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC) TF-IDF filtering query batching generalization patterns sparsity abstraction math and programming abilities cross-lingual generalization role-playing behavior