Measuring Progress in Dictionary Learning for Language Model
  Interpretability with Board Game Models
What latent features are encoded in language model (LM) representations?
Recent work on training sparse autoencoders (SAEs) to disentangle interpretable
features in LM representations has shown significant promise. However,
evaluating the quality of these SAEs is difficult because we lack a
ground-truth collection of interpretable features that we expect good SAEs to
recover. We thus propose to measure progress in interpretable dictionary
learning by working in the setting of LMs trained on chess and Othello
transcripts. These settings carry natural collections of interpretable features
-- for example, "there is a knight on F3" -- which we leverage into
supervised metrics for SAE quality. To guide progress in
interpretable dictionary learning, we introduce a new SAE training technique,
p-annealing, which improves performance on prior unsupervised
metrics as well as our new metrics.
66b0838e2782988785c3f445
Adam Karvonen
extracted_confirmed
2024-08-05T10:41:31.879Z
648b69f2e9baeaafda9cf837
/avatars/cb63703500ab6878b6fd9c3a6c465527.svg
Adam Karvonen
user
adamkarvonen
66b0838e2782988785c3f446
Benjamin Wright
66b0838e2782988785c3f447
Can Rager
admin_assigned
2024-08-05T13:20:58.313Z
66101de68dd1183537d62161
/avatars/d47dd0e4fc5bd08cf4f0c2c1355ea004.svg
Can Rager
user
canrager
66b0838e2782988785c3f448
Rico Angell
66b0838e2782988785c3f449
Jannik Brinkmann
admin_assigned
2024-08-05T13:21:10.461Z
62fe3051029f73cec2766f6f
https://cdn-avatars.huggingface.co/v1/production/uploads/62fe3051029f73cec2766f6f/eEJZhHXcmZGGhV_vCvpTv.jpeg
Jannik Brinkmann
user
jbrinkma
66b0838e2782988785c3f44a
Logan Smith
66b0838e2782988785c3f44b
Claudio Mayrink Verdun
66b0838e2782988785c3f44c
David Bau
admin_assigned
2024-08-05T13:22:00.956Z
6214d6c01e35c843d42d1f77
/avatars/ac208cd180b4f3ed1ec367e581facfcf.svg
David Bau
user
davidbau
66b0838e2782988785c3f44d
Samuel Marks
Measuring Progress in Dictionary Learning for Language Model
  Interpretability with Board Game Models
What latent features are encoded in language model (LM) representations?
Recent work on training sparse autoencoders (SAEs) to disentangle interpretable
features in LM representations has shown significant promise. However,
evaluating the quality of these SAEs is difficult because we lack a
ground-truth collection of interpretable features that we expect good SAEs to
recover. We thus propose to measure progress in interpretable dictionary
learning by working in the setting of LMs trained on chess and Othello
transcripts. These settings carry natural collections of interpretable features
-- for example, "there is a knight on F3" -- which we leverage into
supervised metrics for SAE quality. To guide progress in
interpretable dictionary learning, we introduce a new SAE training technique,
p-annealing, which improves performance on prior unsupervised
metrics as well as our new metrics.
Progress in interpretable dictionary learning for language models is evaluated using supervised metrics derived from chess and Othello transcripts, and enhanced by a new training technique called p-annealing.
sparse autoencoders interpretable features language model representations supervised metrics p-annealing unsupervised metrics