An LLM Feature-based Framework for Dialogue Constructiveness Assessment
<|reference_start|>An LLM Feature-based Framework for Dialogue Constructiveness Assessment: Research on dialogue constructiveness assessment focuses on (i) analysing conversational factors that influence individuals to take specific actions, win debates, change their perspectives or broaden their open-mindedness and (ii) predicting constructiveness outcomes following dialogues for such use cases. These objectives can be achieved by training either interpretable feature-based models (which often involve costly human annotations) or neural models such as pre-trained language models (which have empirically shown higher task accuracy but lack interpretability). In this paper we propose an LLM feature-based framework for dialogue constructiveness assessment that combines the strengths of feature-based and neural approaches, while mitigating their downsides. The framework first defines a set of dataset-independent and interpretable linguistic features, which can be extracted by both prompting an LLM and simple heuristics. Such features are then used to train LLM feature-based models. We apply this framework to three datasets of dialogue constructiveness and find that our LLM feature-based models outperform or performs at least as well as standard feature-based models and neural models. We also find that the LLM feature-based model learns more robust prediction rules instead of relying on superficial shortcuts, which often trouble neural models.<|reference_end|>
@article{zhou2024an,
    title={An LLM Feature-based Framework for Dialogue Constructiveness Assessment},
    author={Lexin Zhou, Youmna Farag and Andreas Vlachos},
    journal={arXiv preprint arXiv:2406.14760},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.14760},
    primaryClass={cs.CL cs.AI cs.LG}
}