title: Measuring memorization in RLHF for code completion
summary: Reinforcement learning with human feedback (RLHF) has become the dominant
  method to align large models to user preferences. Unlike fine-tuning, for which
  there are many studies regarding training data memorization, it is not clear
  how memorization is affected by or introduced in the RLHF alignment process.
  Understanding this relationship is important as real user data may be collected
  and used to align large models; if user data is memorized during RLHF and later
  regurgitated, this could raise privacy concerns. In this work, we analyze how
  training data memorization can surface and propagate through each phase of
  RLHF. We focus our study on code completion models, as code completion is one
  of the most popular use cases for large language models. We find that RLHF
  significantly decreases the chance that data used for reward modeling and
  reinforcement learning is memorized, in comparison to aligning via directly
  fine-tuning on this data, but that examples already memorized during the
  fine-tuning stage of RLHF, will, in the majority of cases, remain memorized
  after RLHF.
paper_authors._id: 6674aa5266539bc0a0e09363
paper_authors.name: Aneesh Pappu
paper_authors._id: 6674aa5266539bc0a0e09364
paper_authors.name: Billy Porter
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-20T22:20:03.982Z
paper_authors._id: 6674a6bea845e4470f065409
paper_authors.avatarUrl: /avatars/d353fbed5294b017c964a726177a4f06.svg
paper_authors.fullname: Billy Porter
paper_authors.type: user
paper_authors.user: billyporter
paper_authors._id: 6674aa5266539bc0a0e09365
paper_authors.name: Ilia Shumailov
paper_authors._id: 6674aa5266539bc0a0e09366
paper_authors.name: Jamie Hayes
paper_title: Measuring memorization in RLHF for code completion
paper_summary: Reinforcement learning with human feedback (RLHF) has become the dominant
  method to align large models to user preferences. Unlike fine-tuning, for which
  there are many studies regarding training data memorization, it is not clear
  how memorization is affected by or introduced in the RLHF alignment process.
  Understanding this relationship is important as real user data may be collected
  and used to align large models; if user data is memorized during RLHF and later
  regurgitated, this could raise privacy concerns. In this work, we analyze how
  training data memorization can surface and propagate through each phase of
  RLHF. We focus our study on code completion models, as code completion is one
  of the most popular use cases for large language models. We find that RLHF
  significantly decreases the chance that data used for reward modeling and
  reinforcement learning is memorized, in comparison to aligning via directly
  fine-tuning on this data, but that examples already memorized during the
  fine-tuning stage of RLHF, will, in the majority of cases, remain memorized
  after RLHF.
paper_ai_keywords: reinforcement learning with human feedback (RLHF) monolithic models user preferences reward modeling training data memorization privacy concerns code completion(models) long language models long language model example memorization code completion models