title: Counterfactual Generation from Language Models
summary: Understanding and manipulating the causal generation mechanisms in language
  models is essential for controlling their behavior. Previous work has primarily
  relied on techniques such as representation surgery -- e.g., model ablations or
  manipulation of linear subspaces tied to specific concepts -- to intervene on
  these models. To understand the impact of interventions precisely, it is useful
  to examine counterfactuals -- e.g., how a given sentence would have appeared
  had it been generated by the model following a specific intervention. We
  highlight that counterfactual reasoning is conceptually distinct from
  interventions, as articulated in Pearl's causal hierarchy. Based on this
  observation, we propose a framework for generating true string counterfactuals
  by reformulating language models as Generalized Structural-equation. Models
  using the Gumbel-max trick. This allows us to model the joint distribution over
  original strings and their counterfactuals resulting from the same
  instantiation of the sampling noise. We develop an algorithm based on hindsight
  Gumbel sampling that allows us to infer the latent noise variables and generate
  counterfactuals of observed strings. Our experiments demonstrate that the
  approach produces meaningful counterfactuals while at the same time showing
  that commonly used intervention techniques have considerable undesired side
  effects.
paper_authors._id: 6732cc097adce85d6b98eda1
paper_authors.name: Shauli Ravfogel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:27:04.233Z
paper_authors._id: 635686ec5aeb69011c7d1abd
paper_authors.avatarUrl: /avatars/c59034ad2c9c2daf4b4a8d3c56449f5e.svg
paper_authors.fullname: Shauli Ravfogel
paper_authors.type: user
paper_authors.user: ravfogs
paper_authors._id: 6732cc097adce85d6b98eda2
paper_authors.name: Anej Svete
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-21T21:33:12.209Z
paper_authors._id: 6629228e0e7290809cd6b243
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/6KfT9MexmALsAywliij5V.jpeg
paper_authors.fullname: Anej Svete
paper_authors.type: user
paper_authors.user: AnejSvete
paper_authors._id: 6732cc097adce85d6b98eda3
paper_authors.name: Vésteinn Snæbjarnarson
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-12T03:31:21.993Z
paper_authors._id: 5f915e09161dc51925934edf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f915e09161dc51925934edf/2PqjbJ7FVG3FOCNmn7ORY.png
paper_authors.fullname: Vésteinn Snæbjarnarson
paper_authors.type: user
paper_authors.user: vesteinn
paper_authors._id: 6732cc097adce85d6b98eda4
paper_authors.name: Ryan Cotterell
paper_title: Counterfactual Generation from Language Models
paper_summary: Understanding and manipulating the causal generation mechanisms in language
  models is essential for controlling their behavior. Previous work has primarily
  relied on techniques such as representation surgery -- e.g., model ablations or
  manipulation of linear subspaces tied to specific concepts -- to intervene on
  these models. To understand the impact of interventions precisely, it is useful
  to examine counterfactuals -- e.g., how a given sentence would have appeared
  had it been generated by the model following a specific intervention. We
  highlight that counterfactual reasoning is conceptually distinct from
  interventions, as articulated in Pearl's causal hierarchy. Based on this
  observation, we propose a framework for generating true string counterfactuals
  by reformulating language models as Generalized Structural-equation. Models
  using the Gumbel-max trick. This allows us to model the joint distribution over
  original strings and their counterfactuals resulting from the same
  instantiation of the sampling noise. We develop an algorithm based on hindsight
  Gumbel sampling that allows us to infer the latent noise variables and generate
  counterfactuals of observed strings. Our experiments demonstrate that the
  approach produces meaningful counterfactuals while at the same time showing
  that commonly used intervention techniques have considerable undesired side
  effects.
paper_ai_keywords: structural-equation models Gumbel-max trick true string counterfactuals hindsight Gumbel sampling latent noise variables