Referring Expression Generation in Visually Grounded Dialogue with Discourse-aware Comprehension Guiding
<|reference_start|>Referring Expression Generation in Visually Grounded Dialogue with Discourse-aware Comprehension Guiding: We propose an approach to referring expression generation (REG) in visually grounded dialogue that is meant to produce referring expressions (REs) that are both discriminative and discourse-appropriate. Our method constitutes a two-stage process. First, we model REG as a text- and image-conditioned next-token prediction task. REs are autoregressively generated based on their preceding linguistic context and a visual representation of the referent. Second, we propose the use of discourse-aware comprehension guiding as part of a generate-and-rerank strategy through which candidate REs generated with our REG model are reranked based on their discourse-dependent discriminatory power. Results from our human evaluation indicate that our proposed two-stage approach is effective in producing discriminative REs, with higher performance in terms of text-image retrieval accuracy for reranked REs compared to those generated using greedy decoding.<|reference_end|>
@article{willemsen2024referring,
    title={Referring Expression Generation in Visually Grounded Dialogue with
  Discourse-aware Comprehension Guiding},
    author={Bram Willemsen, Gabriel Skantze},
    journal={arXiv preprint arXiv:2409.05721},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.05721},
    primaryClass={cs.CL cs.AI cs.CV}
}