title: SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model
    Post-training
summary: Supervised fine-tuning (SFT) and reinforcement learning (RL) are widely used
  post-training techniques for foundation models. However, their roles in
  enhancing model generalization capabilities remain unclear. This paper studies
  the difference between SFT and RL on generalization and memorization, focusing
  on text-based rule variants and visual variants. We introduce GeneralPoints, an
  arithmetic reasoning card game, and adopt V-IRL, a real-world navigation
  environment, to assess how models trained with SFT and RL generalize to unseen
  variants in both textual and visual domains. We show that RL, especially when
  trained with an outcome-based reward, generalizes across both rule-based
  textual and visual variants. SFT, in contrast, tends to memorize training data
  and struggles to generalize out-of-distribution scenarios. Further analysis
  reveals that RL improves the model's underlying visual recognition
  capabilities, contributing to its enhanced generalization in the visual domain.
  Despite RL's superior generalization, we show that SFT remains essential for
  effective RL training; SFT stabilizes the model's output format, enabling
  subsequent RL to achieve its performance gains. These findings demonstrates the
  capability of RL for acquiring generalizable knowledge in complex, multi-modal
  tasks.
paper_authors._id: 6799b39b15f4661561c22968
paper_authors.name: Tianzhe Chu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T17:16:03.469Z
paper_authors._id: 65127ec162b7d28b1eaab17a
paper_authors.avatarUrl: /avatars/0bcfeb68a405be4efb6e8a29738a5598.svg
paper_authors.fullname: Tianzhe
paper_authors.type: user
paper_authors.user: tianzhechu
paper_authors._id: 6799b39b15f4661561c22969
paper_authors.name: Yuexiang Zhai
paper_authors._id: 6799b39b15f4661561c2296a
paper_authors.name: Jihan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:18:57.418Z
paper_authors._id: 6304baf041387c7f1177a5d2
paper_authors.avatarUrl: /avatars/795c63f2394080eec78ca7981d4a1f78.svg
paper_authors.fullname: Jihan Yang
paper_authors.type: user
paper_authors.user: jihanyang
paper_authors._id: 6799b39b15f4661561c2296b
paper_authors.name: Shengbang Tong
paper_authors._id: 6799b39b15f4661561c2296c
paper_authors.name: Saining Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:19:29.194Z
paper_authors._id: 6596422646624a86ff3b3bda
paper_authors.avatarUrl: /avatars/216e12b77e45ac5f1fa20932f5745411.svg
paper_authors.fullname: Saining Xie
paper_authors.type: user
paper_authors.user: sainx
paper_authors._id: 6799b39b15f4661561c2296d
paper_authors.name: Dale Schuurmans
paper_authors._id: 6799b39b15f4661561c2296e
paper_authors.name: Quoc V. Le
paper_authors._id: 6799b39b15f4661561c2296f
paper_authors.name: Sergey Levine
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:19:47.824Z
paper_authors._id: 665ce54120a307a3754849dd
paper_authors.avatarUrl: /avatars/e698726e9be61dd50ce2efe372ed5dac.svg
paper_authors.fullname: Sergey Levine
paper_authors.type: user
paper_authors.user: svlevine
paper_authors._id: 6799b39b15f4661561c22970
paper_authors.name: Yi Ma
paper_title: SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model
    Post-training
paper_summary: Supervised fine-tuning (SFT) and reinforcement learning (RL) are widely used
  post-training techniques for foundation models. However, their roles in
  enhancing model generalization capabilities remain unclear. This paper studies
  the difference between SFT and RL on generalization and memorization, focusing
  on text-based rule variants and visual variants. We introduce GeneralPoints, an
  arithmetic reasoning card game, and adopt V-IRL, a real-world navigation
  environment, to assess how models trained with SFT and RL generalize to unseen
  variants in both textual and visual domains. We show that RL, especially when
  trained with an outcome-based reward, generalizes across both rule-based
  textual and visual variants. SFT, in contrast, tends to memorize training data
  and struggles to generalize out-of-distribution scenarios. Further analysis
  reveals that RL improves the model's underlying visual recognition
  capabilities, contributing to its enhanced generalization in the visual domain.
  Despite RL's superior generalization, we show that SFT remains essential for
  effective RL training; SFT stabilizes the model's output format, enabling
  subsequent RL to achieve its performance gains. These findings demonstrates the
  capability of RL for acquiring generalizable knowledge in complex, multi-modal
  tasks.
paper_ai_summary: Reinforcement learning demonstrates superior generalization compared to supervised fine-tuning across textual and visual domains, while supervised fine-tuning stabilizes the model output format essential for RL performance.
paper_ai_keywords: supervised fine-tuning reinforcement learning generalization memorization text-based rule variants visual variants GeneralPoints V-IRL outcome-based reward visual recognition capabilities multi-modal tasks