title: Insights from the Inverse: Reconstructing LLM Training Goals Through
    Inverse RL
summary: Large language models (LLMs) trained with Reinforcement Learning from Human
  Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying
  reward functions and decision-making processes remain opaque. This paper
  introduces a novel approach to interpreting LLMs by applying inverse
  reinforcement learning (IRL) to recover their implicit reward functions. We
  conduct experiments on toxicity-aligned LLMs of varying sizes, extracting
  reward models that achieve up to 80.40% accuracy in predicting human
  preferences. Our analysis reveals key insights into the non-identifiability of
  reward functions, the relationship between model size and interpretability, and
  potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward
  models can be used to fine-tune new LLMs, resulting in comparable or improved
  performance on toxicity benchmarks. This work provides a new lens for
  understanding and improving LLM alignment, with implications for the
  responsible development and deployment of these powerful systems.
paper_authors._id: 6710e1e2786f6ba453d1787f
paper_authors.name: Jared Joselowitz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:42:56.439Z
paper_authors._id: 63471d1fc849ba78f285a59f
paper_authors.avatarUrl: /avatars/e2445fb517427360e6c2adc8c683c244.svg
paper_authors.fullname: Jared Joselowitz
paper_authors.type: user
paper_authors.user: jaredjoss
paper_authors._id: 6710e1e2786f6ba453d17880
paper_authors.name: Arjun Jagota
paper_authors._id: 6710e1e2786f6ba453d17881
paper_authors.name: Satyapriya Krishna
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:23:44.325Z
paper_authors._id: 6186fef1b1085ab638324e7f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6186fef1b1085ab638324e7f/BL6_WJCkxB-BatBUBilT8.jpeg
paper_authors.fullname: Satya
paper_authors.type: user
paper_authors.user: skrishna
paper_authors._id: 6710e1e2786f6ba453d17882
paper_authors.name: Sonali Parbhoo
paper_title: Insights from the Inverse: Reconstructing LLM Training Goals Through
    Inverse RL
paper_summary: Large language models (LLMs) trained with Reinforcement Learning from Human
  Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying
  reward functions and decision-making processes remain opaque. This paper
  introduces a novel approach to interpreting LLMs by applying inverse
  reinforcement learning (IRL) to recover their implicit reward functions. We
  conduct experiments on toxicity-aligned LLMs of varying sizes, extracting
  reward models that achieve up to 80.40% accuracy in predicting human
  preferences. Our analysis reveals key insights into the non-identifiability of
  reward functions, the relationship between model size and interpretability, and
  potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward
  models can be used to fine-tune new LLMs, resulting in comparable or improved
  performance on toxicity benchmarks. This work provides a new lens for
  understanding and improving LLM alignment, with implications for the
  responsible development and deployment of these powerful systems.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) Large language models (LLMs) inverse reinforcement learning (IRL) reward functions decision-making processes toxicity-aligned LLMs reward models human preferences non-identifiability of reward functions interpretability IRL-derived reward models fine-tune toxicity benchmarks LLM alignment