title: The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context
    Learning
summary: The alignment tuning process of large language models (LLMs) typically
  involves instruction learning through supervised fine-tuning (SFT) and
  preference tuning via reinforcement learning from human feedback (RLHF). A
  recent study, LIMA (Zhou et al. 2023), shows that using merely 1K examples for
  SFT can achieve significant alignment performance as well, suggesting that the
  effect of alignment tuning might be "superficial." This raises questions about
  how exactly the alignment tuning transforms a base LLM.
    We analyze the effect of alignment tuning by examining the token distribution
  shift between base LLMs and their aligned counterpart. Our findings reveal that
  base LLMs and their alignment-tuned versions perform nearly identically in
  decoding on the majority of token positions. Most distribution shifts occur
  with stylistic tokens. These direct evidence strongly supports the Superficial
  Alignment Hypothesis suggested by LIMA.
    Based on these findings, we rethink the alignment of LLMs by posing the
  research question: how effectively can we align base LLMs without SFT or RLHF?
  To address this, we introduce a simple, tuning-free alignment method, URIAL.
  URIAL achieves effective alignment purely through in-context learning (ICL)
  with base LLMs, requiring as few as three constant stylistic examples and a
  system prompt. We conduct a fine-grained and interpretable evaluation on a
  diverse set of examples, named JUST-EVAL-INSTRUCT. Results demonstrate that
  base LLMs with URIAL can match or even surpass the performance of LLMs aligned
  with SFT or SFT+RLHF. We show that the gap between tuning-free and tuning-based
  alignment methods can be significantly reduced through strategic prompting and
  ICL. Our findings on the superficial nature of alignment tuning and results
  with URIAL suggest that deeper analysis and theoretical understanding of
  alignment is crucial to future LLM research.
paper_authors._id: 656e98aad4de03a07d263cc9
paper_authors.name: Bill Yuchen Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-05T08:40:08.311Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_authors._id: 656e98aad4de03a07d263cca
paper_authors.name: Abhilasha Ravichander
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T08:53:27.527Z
paper_authors._id: 6349886c429608888c42319a
paper_authors.avatarUrl: /avatars/f84b5fe8b76172878274754e3399d6ec.svg
paper_authors.fullname: Abhilasha Ravichander
paper_authors.type: user
paper_authors.user: lasha-nlp
paper_authors._id: 656e98aad4de03a07d263ccb
paper_authors.name: Ximing Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T08:53:34.693Z
paper_authors._id: 640928bd3461c51cf7378707
paper_authors.avatarUrl: /avatars/429d1ba8327df82a8f8360c0f55e199c.svg
paper_authors.fullname: Ximing Lu
paper_authors.type: user
paper_authors.user: Ximing
paper_authors._id: 656e98aad4de03a07d263ccc
paper_authors.name: Nouha Dziri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T08:54:05.729Z
paper_authors._id: 625888c2c964680d85d70bff
paper_authors.avatarUrl: /avatars/2ba91e452c5ea577ec9f1f71abc2fa49.svg
paper_authors.fullname: Nouha Dziri
paper_authors.type: user
paper_authors.user: nouhadziri
paper_authors._id: 656e98aad4de03a07d263ccd
paper_authors.name: Melanie Sclar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T08:54:35.585Z
paper_authors._id: 6274d1dba2d1b8d4982e36a8
paper_authors.avatarUrl: /avatars/933654c7b0946a8609aa38cfee364604.svg
paper_authors.fullname: Melanie Sclar
paper_authors.type: user
paper_authors.user: msclar
paper_authors._id: 656e98aad4de03a07d263cce
paper_authors.name: Khyathi Chandu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T08:54:46.023Z
paper_authors._id: 62ebe6472f8999993a847bb6
paper_authors.avatarUrl: /avatars/29fb590f8116bdc9c55f7a333e2c4280.svg
paper_authors.fullname: Khyathi Raghavi Chandu
paper_authors.type: user
paper_authors.user: khyathi
paper_authors._id: 656e98aad4de03a07d263ccf
paper_authors.name: Chandra Bhagavatula
paper_authors._id: 656e98aad4de03a07d263cd0
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T08:55:10.973Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_title: The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context
    Learning
paper_summary: The alignment tuning process of large language models (LLMs) typically
  involves instruction learning through supervised fine-tuning (SFT) and
  preference tuning via reinforcement learning from human feedback (RLHF). A
  recent study, LIMA (Zhou et al. 2023), shows that using merely 1K examples for
  SFT can achieve significant alignment performance as well, suggesting that the
  effect of alignment tuning might be "superficial." This raises questions about
  how exactly the alignment tuning transforms a base LLM.
    We analyze the effect of alignment tuning by examining the token distribution
  shift between base LLMs and their aligned counterpart. Our findings reveal that
  base LLMs and their alignment-tuned versions perform nearly identically in
  decoding on the majority of token positions. Most distribution shifts occur
  with stylistic tokens. These direct evidence strongly supports the Superficial
  Alignment Hypothesis suggested by LIMA.
    Based on these findings, we rethink the alignment of LLMs by posing the
  research question: how effectively can we align base LLMs without SFT or RLHF?
  To address this, we introduce a simple, tuning-free alignment method, URIAL.
  URIAL achieves effective alignment purely through in-context learning (ICL)
  with base LLMs, requiring as few as three constant stylistic examples and a
  system prompt. We conduct a fine-grained and interpretable evaluation on a
  diverse set of examples, named JUST-EVAL-INSTRUCT. Results demonstrate that
  base LLMs with URIAL can match or even surpass the performance of LLMs aligned
  with SFT or SFT+RLHF. We show that the gap between tuning-free and tuning-based
  alignment methods can be significantly reduced through strategic prompting and
  ICL. Our findings on the superficial nature of alignment tuning and results
  with URIAL suggest that deeper analysis and theoretical understanding of
  alignment is crucial to future LLM research.
paper_ai_keywords: supervised fine-tuning (SFT) reinforcement learning from human feedback (RLHF) latent diffusion models (LDMs) token distribution shift decoding stylistic tokens in-context learning (ICL) system prompt JUST-EVAL-INSTRUCT tuning-free alignment method strategic prompting