‡https://ai.meta.com/resources/models-and-libraries/llama/
§We are delaying the release of the 34B model due to a lack of time to sufficiently red team.
¶https://ai.meta.com/llama
‖https://github.com/facebookresearch/llama

4

Figure 4: Training of Llama 2-Chat: This process begins with the pretraining of Llama 2 using publicly
available online sources. Following this, we create an initial version of Llama 2-Chat through the application
of supervised fine-tuning. Subsequently, the model is iteratively refined using Reinforcement Learning
with Human Feedback (RLHF) methodologies, specifically through rejection sampling and Proximal Policy
Optimization (PPO). Throughout the RLHF stage, the accumulation of iterative reward modeling data in
parallel with model enhancements is crucial to ensure the reward models remain within distribution.
Llama2