Multimodal Contextualized Semantic Parsing from Speech
<|reference_start|>Multimodal Contextualized Semantic Parsing from Speech: We introduce Semantic Parsing in Contextual Environments (SPICE), a task designed to enhance artificial agents' contextual awareness by integrating multimodal inputs with prior contexts. SPICE goes beyond traditional semantic parsing by offering a structured, interpretable framework for dynamically updating an agent's knowledge with new information, mirroring the complexity of human communication. We develop the VG-SPICE dataset, crafted to challenge agents with visual scene graph construction from spoken conversational exchanges, highlighting speech and visual data integration. We also present the Audio-Vision Dialogue Scene Parser (AViD-SP) developed for use on VG-SPICE. These innovations aim to improve multimodal information processing and integration. Both the VG-SPICE dataset and the AViD-SP model are publicly available.<|reference_end|>
@article{voas2024multimodal,
    title={Multimodal Contextualized Semantic Parsing from Speech},
    author={Jordan Voas, Raymond Mooney, and David Harwath},
    journal={arXiv preprint arXiv:2406.06438},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.06438},
    primaryClass={cs.CL cs.CV cs.HC cs.LG cs.SD eess.AS}
}