title: Reasons to Reject? Aligning Language Models with Judgments
summary: As humans, we consistently engage in interactions with our peers and receive
  feedback in the form of natural language. This language feedback allows us to
  reflect on our actions, maintain appropriate behavior, and rectify our errors.
  The question arises naturally: can we use language feedback to align large
  language models (LLMs)? In contrast to previous research that aligns LLMs with
  reward or preference data, we present the first systematic exploration of
  alignment through the lens of language feedback (i.e., judgment). We commence
  with an in-depth investigation of potential methods that can be adapted for
  aligning LLMs with judgments, revealing that these methods are unable to fully
  capitalize on the judgments. To facilitate more effective utilization of
  judgments, we propose a novel framework, Contrastive Unlikelihood Training
  (CUT), that allows for fine-grained inappropriate content detection and
  correction based on judgments. Our offline alignment results show that, with
  merely 1317 off-the-shelf judgment data, CUT (LLaMA2-13b) can beat the 175B
  DaVinci003 and surpass the best baseline by 52.34 points on AlpacaEval. The
  online alignment results demonstrate that CUT can align LLMs (LLaMA2-chat-13b)
  in an iterative fashion using model-specific judgment data, with a steady
  performance improvement from 81.09 to 91.36 points on AlpacaEval. Our analysis
  further suggests that judgments exhibit greater potential than rewards for LLM
  alignment and warrant future research.
paper_authors._id: 658a49e94f349f95cffb27d3
paper_authors.name: Weiwen Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-26T10:36:59.141Z
paper_authors._id: 64118689756b9e455c7eac62
paper_authors.avatarUrl: /avatars/cdb3da22593facf545a0bafbf548b07e.svg
paper_authors.fullname: Xu Weiwen
paper_authors.type: user
paper_authors.user: xww033
paper_authors._id: 658a49e94f349f95cffb27d4
paper_authors.name: Deng Cai
paper_authors._id: 658a49e94f349f95cffb27d5
paper_authors.name: Zhisong Zhang
paper_authors._id: 658a49e94f349f95cffb27d6
paper_authors.name: Wai Lam
paper_authors._id: 658a49e94f349f95cffb27d7
paper_authors.name: Shuming Shi
paper_title: Reasons to Reject? Aligning Language Models with Judgments
paper_summary: As humans, we consistently engage in interactions with our peers and receive
  feedback in the form of natural language. This language feedback allows us to
  reflect on our actions, maintain appropriate behavior, and rectify our errors.
  The question arises naturally: can we use language feedback to align large
  language models (LLMs)? In contrast to previous research that aligns LLMs with
  reward or preference data, we present the first systematic exploration of
  alignment through the lens of language feedback (i.e., judgment). We commence
  with an in-depth investigation of potential methods that can be adapted for
  aligning LLMs with judgments, revealing that these methods are unable to fully
  capitalize on the judgments. To facilitate more effective utilization of
  judgments, we propose a novel framework, Contrastive Unlikelihood Training
  (CUT), that allows for fine-grained inappropriate content detection and
  correction based on judgments. Our offline alignment results show that, with
  merely 1317 off-the-shelf judgment data, CUT (LLaMA2-13b) can beat the 175B
  DaVinci003 and surpass the best baseline by 52.34 points on AlpacaEval. The
  online alignment results demonstrate that CUT can align LLMs (LLaMA2-chat-13b)
  in an iterative fashion using model-specific judgment data, with a steady
  performance improvement from 81.09 to 91.36 points on AlpacaEval. Our analysis
  further suggests that judgments exhibit greater potential than rewards for LLM
  alignment and warrant future research.
paper_ai_keywords: large language models (LLMs) Contrastive Unlikelihood Training (CUT) off-the-shelf judgment data inappropriate content detection AlpacaEval iterative fashion model-specific judgment data