Studying Large Language Model Generalization with Influence Functions
When trying to gain better visibility into a machine learning model in order
to understand and mitigate the associated risks, a potentially valuable source
of evidence is: which training examples most contribute to a given behavior?
Influence functions aim to answer a counterfactual: how would the model's
parameters (and hence its outputs) change if a given sequence were added to the
training set? While influence functions have produced insights for small
models, they are difficult to scale to large language models (LLMs) due to the
difficulty of computing an inverse-Hessian-vector product (IHVP). We use the
Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC)
approximation to scale influence functions up to LLMs with up to 52 billion
parameters. In our experiments, EK-FAC achieves similar accuracy to traditional
influence function estimators despite the IHVP computation being orders of
magnitude faster. We investigate two algorithmic techniques to reduce the cost
of computing gradients of candidate training sequences: TF-IDF filtering and
query batching. We use influence functions to investigate the generalization
patterns of LLMs, including the sparsity of the influence patterns, increasing
abstraction with scale, math and programming abilities, cross-lingual
generalization, and role-playing behavior. Despite many apparently
sophisticated forms of generalization, we identify a surprising limitation:
influences decay to near-zero when the order of key phrases is flipped.
Overall, influence functions give us a powerful new tool for studying the
generalization properties of LLMs.
64d1ac192f1f9578a0661d59
Roger Grosse
64d1ac192f1f9578a0661d5a
Juhan Bae
64d1ac192f1f9578a0661d5b
Cem Anil
admin_assigned
2023-08-08T15:03:38.323Z
62f6b27d05ca68c0e0008549
/avatars/14d0075aa1b578cd7ee5f9e68d12e2f0.svg
Cem Anil
user
anilcem
64d1ac192f1f9578a0661d5c
Nelson Elhage
64d1ac192f1f9578a0661d5d
Alex Tamkin
admin_assigned
2023-08-08T15:05:31.550Z
5fc18c42ea82dd667bb0ffaf
/avatars/ae98f82442b560de247d2354cc364398.svg
Alex Tamkin
user
alextamkin
64d1ac192f1f9578a0661d5e
Amirhossein Tajdini
64d1ac192f1f9578a0661d5f
Benoit Steiner
64d1ac192f1f9578a0661d60
Dustin Li
64d1ac192f1f9578a0661d61
Esin Durmus
admin_assigned
2023-08-08T15:06:01.415Z
627dd7dbba19b19eff6aaa46
/avatars/6e43025ccb96e48339d1db477da043b0.svg
Esin Durmus
user
esdurmus
64d1ac192f1f9578a0661d62
Ethan Perez
64d1ac192f1f9578a0661d63
Evan Hubinger
64d1ac192f1f9578a0661d64
Kamilė Lukošiūtė
64d1ac192f1f9578a0661d65
Karina Nguyen
admin_assigned
2023-08-08T15:06:37.243Z
62ef4469cbe9a53fcd5a7f03
/avatars/13352152a67ef304ba444cbc310be5dd.svg
Karina Nguyen
user
karinanguyen
64d1ac192f1f9578a0661d66
Nicholas Joseph
64d1ac192f1f9578a0661d67
Sam McCandlish
64d1ac192f1f9578a0661d68
Jared Kaplan
admin_assigned
2023-08-08T15:07:02.324Z
633b051c7af633cbcd01b0ff
/avatars/e2f315a8d6241ae1a783ba2957a211a5.svg
Jared Kaplan
user
FrizzleFried
64d1ac192f1f9578a0661d69
Samuel R. Bowman
Studying Large Language Model Generalization with Influence Functions
When trying to gain better visibility into a machine learning model in order
to understand and mitigate the associated risks, a potentially valuable source
of evidence is: which training examples most contribute to a given behavior?
Influence functions aim to answer a counterfactual: how would the model's
parameters (and hence its outputs) change if a given sequence were added to the
training set? While influence functions have produced insights for small
models, they are difficult to scale to large language models (LLMs) due to the
difficulty of computing an inverse-Hessian-vector product (IHVP). We use the
Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC)
approximation to scale influence functions up to LLMs with up to 52 billion
parameters. In our experiments, EK-FAC achieves similar accuracy to traditional
influence function estimators despite the IHVP computation being orders of
magnitude faster. We investigate two algorithmic techniques to reduce the cost
of computing gradients of candidate training sequences: TF-IDF filtering and
query batching. We use influence functions to investigate the generalization
patterns of LLMs, including the sparsity of the influence patterns, increasing
abstraction with scale, math and programming abilities, cross-lingual
generalization, and role-playing behavior. Despite many apparently
sophisticated forms of generalization, we identify a surprising limitation:
influences decay to near-zero when the order of key phrases is flipped.
Overall, influence functions give us a powerful new tool for studying the
generalization properties of LLMs.
Influence functions, using the EK-FAC approximation, provide insights into the generalization patterns and limitations of large language models by analyzing the contribution of training examples.
influence functions Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC) inverse-Hessian-vector product (IHVP) TF-IDF filtering query batching generalization patterns sparsity abstraction math and programming abilities cross-lingual generalization role-playing behavior