SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model
  Post-training
Supervised fine-tuning (SFT) and reinforcement learning (RL) are widely used
post-training techniques for foundation models. However, their roles in
enhancing model generalization capabilities remain unclear. This paper studies
the difference between SFT and RL on generalization and memorization, focusing
on text-based rule variants and visual variants. We introduce GeneralPoints, an
arithmetic reasoning card game, and adopt V-IRL, a real-world navigation
environment, to assess how models trained with SFT and RL generalize to unseen
variants in both textual and visual domains. We show that RL, especially when
trained with an outcome-based reward, generalizes across both rule-based
textual and visual variants. SFT, in contrast, tends to memorize training data
and struggles to generalize out-of-distribution scenarios. Further analysis
reveals that RL improves the model's underlying visual recognition
capabilities, contributing to its enhanced generalization in the visual domain.
Despite RL's superior generalization, we show that SFT remains essential for
effective RL training; SFT stabilizes the model's output format, enabling
subsequent RL to achieve its performance gains. These findings demonstrates the
capability of RL for acquiring generalizable knowledge in complex, multi-modal
tasks.
6799b39b15f4661561c22968
Tianzhe Chu
claimed_verified
2025-01-29T17:16:03.469Z
65127ec162b7d28b1eaab17a
/avatars/0bcfeb68a405be4efb6e8a29738a5598.svg
Tianzhe
user
tianzhechu
6799b39b15f4661561c22969
Yuexiang Zhai
6799b39b15f4661561c2296a
Jihan Yang
admin_assigned
2025-01-29T17:18:57.418Z
6304baf041387c7f1177a5d2
/avatars/795c63f2394080eec78ca7981d4a1f78.svg
Jihan Yang
user
jihanyang
6799b39b15f4661561c2296b
Shengbang Tong
6799b39b15f4661561c2296c
Saining Xie
admin_assigned
2025-01-29T17:19:29.194Z
6596422646624a86ff3b3bda
/avatars/216e12b77e45ac5f1fa20932f5745411.svg
Saining Xie
user
sainx
6799b39b15f4661561c2296d
Dale Schuurmans
6799b39b15f4661561c2296e
Quoc V. Le
6799b39b15f4661561c2296f
Sergey Levine
admin_assigned
2025-01-29T17:19:47.824Z
665ce54120a307a3754849dd
/avatars/e698726e9be61dd50ce2efe372ed5dac.svg
Sergey Levine
user
svlevine
6799b39b15f4661561c22970
Yi Ma
SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model
  Post-training
Supervised fine-tuning (SFT) and reinforcement learning (RL) are widely used
post-training techniques for foundation models. However, their roles in
enhancing model generalization capabilities remain unclear. This paper studies
the difference between SFT and RL on generalization and memorization, focusing
on text-based rule variants and visual variants. We introduce GeneralPoints, an
arithmetic reasoning card game, and adopt V-IRL, a real-world navigation
environment, to assess how models trained with SFT and RL generalize to unseen
variants in both textual and visual domains. We show that RL, especially when
trained with an outcome-based reward, generalizes across both rule-based
textual and visual variants. SFT, in contrast, tends to memorize training data
and struggles to generalize out-of-distribution scenarios. Further analysis
reveals that RL improves the model's underlying visual recognition
capabilities, contributing to its enhanced generalization in the visual domain.
Despite RL's superior generalization, we show that SFT remains essential for
effective RL training; SFT stabilizes the model's output format, enabling
subsequent RL to achieve its performance gains. These findings demonstrates the
capability of RL for acquiring generalizable knowledge in complex, multi-modal
tasks.
Reinforcement learning demonstrates superior generalization compared to supervised fine-tuning across textual and visual domains, while supervised fine-tuning stabilizes the model output format essential for RL performance.
supervised fine-tuning reinforcement learning generalization memorization text-based rule variants visual variants GeneralPoints V-IRL outcome-based reward visual recognition capabilities multi-modal tasks