Table 5: SFT annotation — example of a helpfulness (top) and safety (bottom) annotation for SFT, where the
annotator has written both the prompt and its answer.
Getting Started. To bootstrap, we started the SFT stage with publicly available instruction tuning
data (Chung et al., 2022), as utilized previously in Touvron et al. (2023).
Quality Is All You Need. Third-party SFT data is available from many different sources, but we found that
many of these have insufficient diversity and quality — in particular for aligning LLMs towards dialogue-style
instructions. As a result, we focused first on collecting several thousand examples of high-quality SFT data,
as illustrated in Table 5. By setting aside millions of examples from third-party datasets and using fewer but
higher-quality examples from our own vendor-based annotation efforts, our results notably improved. These
findings are similar in spirit to Zhou et al. (2023), which also finds that a limited set of clean instruction-tuning
Llama2