Backtracing: Retrieving the Cause of the Query
Many online content portals allow users to ask questions to supplement their
understanding (e.g., of lectures). While information retrieval (IR) systems may
provide answers for such user queries, they do not directly assist content
creators -- such as lecturers who want to improve their content -- identify
segments that _caused_ a user to ask those questions. We introduce the task of
backtracing, in which systems retrieve the text segment that most likely caused
a user query. We formalize three real-world domains for which backtracing is
important in improving content delivery and communication: understanding the
cause of (a) student confusion in the Lecture domain, (b) reader curiosity in
the News Article domain, and (c) user emotion in the Conversation domain. We
evaluate the zero-shot performance of popular information retrieval methods and
language modeling methods, including bi-encoder, re-ranking and
likelihood-based methods and ChatGPT. While traditional IR systems retrieve
semantically relevant information (e.g., details on "projection matrices" for a
query "does projecting multiple times still lead to the same point?"), they
often miss the causally relevant context (e.g., the lecturer states "projecting
twice gets me the same answer as one projection"). Our results show that there
is room for improvement on backtracing and it requires new retrieval
approaches. We hope our benchmark serves to improve future retrieval systems
for backtracing, spawning systems that refine content generation and identify
linguistic triggers influencing user queries. Our code and data are
open-sourced: https://github.com/rosewang2008/backtracing.
65e958ef063e16f1c3fa7097
Rose E. Wang
admin_assigned
2024-03-07T10:04:03.149Z
62e2047491d6d37b6bff4dfb
/avatars/1c758de8d531aa17e516aa11c6102f4a.svg
Rose E Wang
user
rewang
65e958ef063e16f1c3fa7098
Pawan Wirawarn
65e958ef063e16f1c3fa7099
Omar Khattab
admin_assigned
2024-03-07T10:04:18.338Z
5f764762f1e7ef6e919a1e0f
/avatars/9445f6efa3f78a870e338988d9ce5573.svg
Omar Khattab
user
okhattab
65e958ef063e16f1c3fa709a
Noah Goodman
claimed_verified
2024-11-15T09:49:40.433Z
67321274c1f20c742bcf7a8d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ltcQhre6eDRVzn6Vbbyhu.png
Noah D. Goodman
user
ngoodman
65e958ef063e16f1c3fa709b
Dorottya Demszky
extracted_confirmed
2024-03-07T17:39:15.210Z
606e3a4c7f3150d181025dd1
/avatars/224ee859c1df3f6c4ab59910f2a4732b.svg
Dora Demszky
user
ddemszky
Backtracing: Retrieving the Cause of the Query
Many online content portals allow users to ask questions to supplement their
understanding (e.g., of lectures). While information retrieval (IR) systems may
provide answers for such user queries, they do not directly assist content
creators -- such as lecturers who want to improve their content -- identify
segments that _caused_ a user to ask those questions. We introduce the task of
backtracing, in which systems retrieve the text segment that most likely caused
a user query. We formalize three real-world domains for which backtracing is
important in improving content delivery and communication: understanding the
cause of (a) student confusion in the Lecture domain, (b) reader curiosity in
the News Article domain, and (c) user emotion in the Conversation domain. We
evaluate the zero-shot performance of popular information retrieval methods and
language modeling methods, including bi-encoder, re-ranking and
likelihood-based methods and ChatGPT. While traditional IR systems retrieve
semantically relevant information (e.g., details on "projection matrices" for a
query "does projecting multiple times still lead to the same point?"), they
often miss the causally relevant context (e.g., the lecturer states "projecting
twice gets me the same answer as one projection"). Our results show that there
is room for improvement on backtracing and it requires new retrieval
approaches. We hope our benchmark serves to improve future retrieval systems
for backtracing, spawning systems that refine content generation and identify
linguistic triggers influencing user queries. Our code and data are
open-sourced: https://github.com/rosewang2008/backtracing.
The backtracing task aims to retrieve specific text segments causing user queries, enhancing content creation by identifying confusion, curiosity, and emotional triggers, outperforming traditional information retrieval methods.
backtracing information retrieval bi-encoder re-ranking likelihood-based methods ChatGPT student confusion reader curiosity user emotion linguistic triggers