ChuXin: 16B Technical Report
<|reference_start|>ChuXin: 16B Technical Report: In this report, we present ChuXin, an entirely open-source language model with a size of 1.6 billion parameters. Unlike the majority of works that only open-sourced the model weights and architecture, we have made everything needed to train a model available, including the training data, the training process, and the evaluation code. Our goal is to empower and strengthen the open research community, fostering transparency and enabling a new wave of innovation in the field of language modeling. Furthermore, we extend the context length to 1M tokens through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. The weights for both models are available at Hugging Face to download and use.<|reference_end|>
@article{zhuang2024chuxin:,
    title={ChuXin: 1.6B Technical Report},
    author={Xiaomin Zhuang, Yufan Jiang, Qiaozhi He, Zhihua Wu},
    journal={arXiv preprint arXiv:2405.04828},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.04828},
    primaryClass={cs.CL}
}