360Zhinao Technical Report
<|reference_start|>360Zhinao Technical Report: We present 360Zhinao models with 7B parameter size and context lengths spanning 4K, 32K and 360K, all available at https://github.com/Qihoo360/360zhinao. For rapid development in pretraining, we establish a stable and sensitive ablation environment to evaluate and compare experiment runs with minimal model size. Under such guidance, we perfect our data cleaning and composition strategies to pretrain $\texttt{360Zhinao-7B-Base}$ on 3.4T tokens. We also mainly emphasize data during alignment, where we strive to balance quantity and quality with filtering and reformatting. With tailored data, 360Zhinao-7B's context window is easily extended to 32K and 360K. RMs and RLHF are trained following SFT and credibly applied to specific tasks. All together these contributions lead to 360Zhinao-7B's competitive performance among models of similar size.<|reference_end|>
@article{360zhinao team2024360zhinao,
    title={360Zhinao Technical Report},
    author={360Zhinao Team},
    journal={arXiv preprint arXiv:2405.13386},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.13386},
    primaryClass={cs.CL cs.AI}
}