ReNeLiB: Real-time Neural Listening Behavior Generation for Socially Interactive Agents
<|reference_start|>ReNeLiB: Real-time Neural Listening Behavior Generation for Socially Interactive Agents: Flexible and natural nonverbal reactions to human behavior remain a challenge for socially interactive agents (SIAs) that are predominantly animated using hand-crafted rules. While recently proposed machine learning based approaches to conversational behavior generation are a promising way to address this challenge, they have not yet been employed in SIAs. The primary reason for this is the lack of a software toolkit integrating such approaches with SIA frameworks that conforms to the challenging real-time requirements of human-agent interaction scenarios. In our work, we for the first time present such a toolkit consisting of three main components: (1) real-time feature extraction capturing multi-modal social cues from the user; (2) behavior generation based on a recent state-of-the-art neural network approach; (3) visualization of the generated behavior supporting both FLAME-based and Apple ARKit-based interactive agents. We comprehensively evaluate the real-time performance of the whole framework and its components. In addition, we introduce pre-trained behavioral generation models derived from psychotherapy sessions for domain-specific listening behaviors. Our software toolkit, pivotal for deploying and assessing SIAs' listening behavior in real-time, is publicly available. Resources, including code, behavioural multi-modal features extracted from therapeutic interactions, are hosted at https://daksitha.github.io/ReNeLib<|reference_end|>
@article{don2024renelib:,
    title={ReNeLiB: Real-time Neural Listening Behavior Generation for Socially
  Interactive Agents},
    author={Daksitha Withanage Don, Philipp M"uller, Fabrizio Nunnari, Elisabeth
  Andr'e, Patrick Gebhard},
    journal={arXiv preprint arXiv:2402.08079},
    year={2024},
    doi={10.1145/3577190.3614133},
    archivePrefix={arXiv},
    eprint={2402.08079},
    primaryClass={cs.HC}
}