to preference-based annotation for RLHF.
Fine-Tuning Details. For supervised fine-tuning, we use a cosine learning rate schedule with an initial
learning rate of 2 × 10−5, a weight decay of 0.1, a batch size of 64, and a sequence length of 4096 tokens.
For the fine-tuning process, each sample consists of a prompt and an answer. To ensure the model sequence
length is properly filled, we concatenate all the prompts and answers from the training set. A special token is
utilized to separate the prompt and answer segments. We utilize an autoregressive objective and zero-out
the loss on tokens from the user prompt, so as a result, we backpropagate only on answer tokens. Finally, we
fine-tune the model for 2 epochs.
Llama2