WangchanLion and WangchanX MRC Eval
<|reference_start|>WangchanLion and WangchanX MRC Eval: This technical report describes the development of WangchanLion, an instruction fine-tuned model focusing on Machine Reading Comprehension (MRC) in the Thai language. Our model is based on SEA-LION and a collection of instruction following datasets. To promote open research and reproducibility, we publicly release all training data, code, and the final model weights under the Apache-2 license. To assess the contextual understanding capability, we conducted extensive experimental studies using two Thai MRC datasets, XQuAD and Iapp_wiki_qa_squad. Experimental results demonstrate the model's ability to comprehend the context and produce an answer faithful to the reference one in 0-shot and 1-shot settings. In addition, our evaluation goes beyond the traditional MRC. We propose a new evaluation scheme assessing the answer's correctness, helpfulness, conciseness, and contextuality. Our code is available publicly at https://github.com/vistec-AI/WangchanLion.<|reference_end|>
@article{phatthiyaphaibun2024wangchanlion,
    title={WangchanLion and WangchanX MRC Eval},
    author={Wannaphong Phatthiyaphaibun, Surapon Nonesung, Patomporn
  Payoungkhamdee, Peerat Limkonchotiwat, Can Udomcharoenchaikit, Jitkapat
  Sawatphol, Chompakorn Chaksangchaichot, Ekapol Chuangsuwanich, Sarana
  Nutanong},
    journal={arXiv preprint arXiv:2403.16127},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.16127},
    primaryClass={cs.CL cs.AI}
}