The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context
  Learning
The alignment tuning process of large language models (LLMs) typically
involves instruction learning through supervised fine-tuning (SFT) and
preference tuning via reinforcement learning from human feedback (RLHF). A
recent study, LIMA (Zhou et al. 2023), shows that using merely 1K examples for
SFT can achieve significant alignment performance as well, suggesting that the
effect of alignment tuning might be "superficial." This raises questions about
how exactly the alignment tuning transforms a base LLM.
  We analyze the effect of alignment tuning by examining the token distribution
shift between base LLMs and their aligned counterpart. Our findings reveal that
base LLMs and their alignment-tuned versions perform nearly identically in
decoding on the majority of token positions. Most distribution shifts occur
with stylistic tokens. These direct evidence strongly supports the Superficial
Alignment Hypothesis suggested by LIMA.
  Based on these findings, we rethink the alignment of LLMs by posing the
research question: how effectively can we align base LLMs without SFT or RLHF?
To address this, we introduce a simple, tuning-free alignment method, URIAL.
URIAL achieves effective alignment purely through in-context learning (ICL)
with base LLMs, requiring as few as three constant stylistic examples and a
system prompt. We conduct a fine-grained and interpretable evaluation on a
diverse set of examples, named JUST-EVAL-INSTRUCT. Results demonstrate that
base LLMs with URIAL can match or even surpass the performance of LLMs aligned
with SFT or SFT+RLHF. We show that the gap between tuning-free and tuning-based
alignment methods can be significantly reduced through strategic prompting and
ICL. Our findings on the superficial nature of alignment tuning and results
with URIAL suggest that deeper analysis and theoretical understanding of
alignment is crucial to future LLM research.
656e98aad4de03a07d263cc9
Bill Yuchen Lin
claimed_verified
2023-12-05T08:40:08.311Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
656e98aad4de03a07d263cca
Abhilasha Ravichander
admin_assigned
2023-12-05T08:53:27.527Z
6349886c429608888c42319a
/avatars/f84b5fe8b76172878274754e3399d6ec.svg
Abhilasha Ravichander
user
lasha-nlp
656e98aad4de03a07d263ccb
Ximing Lu
admin_assigned
2023-12-05T08:53:34.693Z
640928bd3461c51cf7378707
/avatars/b29fcb7388b81f8686086352f6321d06.svg
Ximing Lu
user
Ximing
656e98aad4de03a07d263ccc
Nouha Dziri
admin_assigned
2023-12-05T08:54:05.729Z
625888c2c964680d85d70bff
/avatars/2ba91e452c5ea577ec9f1f71abc2fa49.svg
Nouha Dziri
user
nouhadziri
656e98aad4de03a07d263ccd
Melanie Sclar
admin_assigned
2023-12-05T08:54:35.585Z
6274d1dba2d1b8d4982e36a8
/avatars/933654c7b0946a8609aa38cfee364604.svg
Melanie Sclar
user
msclar
656e98aad4de03a07d263cce
Khyathi Chandu
admin_assigned
2023-12-05T08:54:46.023Z
62ebe6472f8999993a847bb6
/avatars/29fb590f8116bdc9c55f7a333e2c4280.svg
Khyathi Raghavi Chandu
user
khyathi
656e98aad4de03a07d263ccf
Chandra Bhagavatula
656e98aad4de03a07d263cd0
Yejin Choi
admin_assigned
2023-12-05T08:55:10.973Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context
  Learning
The alignment tuning process of large language models (LLMs) typically
involves instruction learning through supervised fine-tuning (SFT) and
preference tuning via reinforcement learning from human feedback (RLHF). A
recent study, LIMA (Zhou et al. 2023), shows that using merely 1K examples for
SFT can achieve significant alignment performance as well, suggesting that the
effect of alignment tuning might be "superficial." This raises questions about
how exactly the alignment tuning transforms a base LLM.
  We analyze the effect of alignment tuning by examining the token distribution
shift between base LLMs and their aligned counterpart. Our findings reveal that
base LLMs and their alignment-tuned versions perform nearly identically in
decoding on the majority of token positions. Most distribution shifts occur
with stylistic tokens. These direct evidence strongly supports the Superficial
Alignment Hypothesis suggested by LIMA.
  Based on these findings, we rethink the alignment of LLMs by posing the
research question: how effectively can we align base LLMs without SFT or RLHF?
To address this, we introduce a simple, tuning-free alignment method, URIAL.
URIAL achieves effective alignment purely through in-context learning (ICL)
with base LLMs, requiring as few as three constant stylistic examples and a
system prompt. We conduct a fine-grained and interpretable evaluation on a
diverse set of examples, named JUST-EVAL-INSTRUCT. Results demonstrate that
base LLMs with URIAL can match or even surpass the performance of LLMs aligned
with SFT or SFT+RLHF. We show that the gap between tuning-free and tuning-based
alignment methods can be significantly reduced through strategic prompting and
ICL. Our findings on the superficial nature of alignment tuning and results
with URIAL suggest that deeper analysis and theoretical understanding of
alignment is crucial to future LLM research.
A study explores the effectiveness of alignment tuning in large language models, suggesting that superficial methods like in-context learning can achieve strong performance without traditional supervised fine-tuning or reinforcement learning.
instruction learning supervised fine-tuning reinforcement learning from human feedback token distribution shift stylistic tokens Superficial Alignment Hypothesis URIAL in-context learning JUST-EVAL-INSTRUCT tuning-free alignment