CNIMA: A Universal Evaluation Framework and Automated Approach for Assessing Second Language Dialogues
<|reference_start|>CNIMA: A Universal Evaluation Framework and Automated Approach for Assessing Second Language Dialogues: We develop CNIMA (Chinese Non-Native Interactivity Measurement and Automation), a Chinese-as-a-second-language labelled dataset with 10K dialogues. We annotate CNIMA using an evaluation framework -- originally introduced for English-as-a-second-language dialogues -- that assesses micro-level features (e.g.\ backchannels) and macro-level interactivity labels (e.g.\ topic management) and test the framework's transferability from English to Chinese. We found the framework robust across languages and revealed universal and language-specific relationships between micro-level and macro-level features. Next, we propose an approach to automate the evaluation and find strong performance, creating a new tool for automated second language assessment. Our system can be adapted to other languages easily as it uses large language models and as such does not require large-scale annotated training data.<|reference_end|>
@article{gao2024cnima:,
    title={CNIMA: A Universal Evaluation Framework and Automated Approach for
  Assessing Second Language Dialogues},
    author={Rena Gao, Jingxuan Wu, Carsten Roever, Xuetong Wu, Jing Wu, Long Lv,
  Jey Han Lau},
    journal={arXiv preprint arXiv:2408.16518},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.16518},
    primaryClass={cs.CL}
}