Persona Vectors: Monitoring and Controlling Character Traits in Language
  Models
Large language models interact with users through a simulated 'Assistant'
persona. While the Assistant is typically trained to be helpful, harmless, and
honest, it sometimes deviates from these ideals. In this paper, we identify
directions in the model's activation space-persona vectors-underlying several
traits, such as evil, sycophancy, and propensity to hallucinate. We confirm
that these vectors can be used to monitor fluctuations in the Assistant's
personality at deployment time. We then apply persona vectors to predict and
control personality shifts that occur during training. We find that both
intended and unintended personality changes after finetuning are strongly
correlated with shifts along the relevant persona vectors. These shifts can be
mitigated through post-hoc intervention, or avoided in the first place with a
new preventative steering method. Moreover, persona vectors can be used to flag
training data that will produce undesirable personality changes, both at the
dataset level and the individual sample level. Our method for extracting
persona vectors is automated and can be applied to any personality trait of
interest, given only a natural-language description.
68898d7531e1218a089281b4
Runjin Chen
68898d7531e1218a089281b5
Andy Arditi
68898d7531e1218a089281b6
Henry Sleight
68898d7531e1218a089281b7
Owain Evans
68898d7531e1218a089281b8
Jack Lindsey
Persona Vectors: Monitoring and Controlling Character Traits in Language
  Models
Large language models interact with users through a simulated 'Assistant'
persona. While the Assistant is typically trained to be helpful, harmless, and
honest, it sometimes deviates from these ideals. In this paper, we identify
directions in the model's activation space-persona vectors-underlying several
traits, such as evil, sycophancy, and propensity to hallucinate. We confirm
that these vectors can be used to monitor fluctuations in the Assistant's
personality at deployment time. We then apply persona vectors to predict and
control personality shifts that occur during training. We find that both
intended and unintended personality changes after finetuning are strongly
correlated with shifts along the relevant persona vectors. These shifts can be
mitigated through post-hoc intervention, or avoided in the first place with a
new preventative steering method. Moreover, persona vectors can be used to flag
training data that will produce undesirable personality changes, both at the
dataset level and the individual sample level. Our method for extracting
persona vectors is automated and can be applied to any personality trait of
interest, given only a natural-language description.
Persona vectors in large language models can monitor and control personality changes during training and deployment, enabling the identification and mitigation of undesirable traits.
activation space persona vectors evil sycophancy hallucination personality shifts finetuning post-hoc intervention preventative steering method training data natural-language description