title: TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long
    Documents
summary: Large language models (LLMs) have attracted huge interest in practical
  applications given their increasingly accurate responses and coherent reasoning
  abilities. Given their nature as black-boxes using complex reasoning processes
  on their inputs, it is inevitable that the demand for scalable and faithful
  explanations for LLMs' generated content will continue to grow. There have been
  major developments in the explainability of neural network models over the past
  decade. Among them, post-hoc explainability methods, especially Shapley values,
  have proven effective for interpreting deep learning models. However, there are
  major challenges in scaling up Shapley values for LLMs, particularly when
  dealing with long input contexts containing thousands of tokens and
  autoregressively generated output sequences. Furthermore, it is often unclear
  how to effectively utilize generated explanations to improve the performance of
  LLMs. In this paper, we introduce TextGenSHAP, an efficient post-hoc
  explanation method incorporating LM-specific techniques. We demonstrate that
  this leads to significant increases in speed compared to conventional Shapley
  value computations, reducing processing times from hours to minutes for
  token-level explanations, and to just seconds for document-level explanations.
  In addition, we demonstrate how real-time Shapley values can be utilized in two
  important scenarios, providing better understanding of long-document question
  answering by localizing important words and sentences; and improving existing
  document retrieval systems through enhancing the accuracy of selected passages
  and ultimately the final responses.
paper_authors._id: 656e96b62cd7a3e348fff821
paper_authors.name: James Enouen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T12:24:50.130Z
paper_authors._id: 64fd49f70e486522f85f0bc2
paper_authors.avatarUrl: /avatars/71ac3b7b5111aafb465062bb408145cd.svg
paper_authors.fullname: James Enouen
paper_authors.type: user
paper_authors.user: 4Z0E50Xg5pQ
paper_authors._id: 656e96b62cd7a3e348fff822
paper_authors.name: Hootan Nakhost
paper_authors._id: 656e96b62cd7a3e348fff823
paper_authors.name: Sayna Ebrahimi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T12:25:03.197Z
paper_authors._id: 647e51337f9ad5e44baafe68
paper_authors.avatarUrl: /avatars/264715eaeec7a28a9fea89f12732d8a3.svg
paper_authors.fullname: Sayna Ebrahimi
paper_authors.type: user
paper_authors.user: SaynaEbrahimi
paper_authors._id: 656e96b62cd7a3e348fff824
paper_authors.name: Sercan O Arik
paper_authors._id: 656e96b62cd7a3e348fff825
paper_authors.name: Yan Liu
paper_authors._id: 656e96b62cd7a3e348fff826
paper_authors.name: Tomas Pfister
paper_title: TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long
    Documents
paper_summary: Large language models (LLMs) have attracted huge interest in practical
  applications given their increasingly accurate responses and coherent reasoning
  abilities. Given their nature as black-boxes using complex reasoning processes
  on their inputs, it is inevitable that the demand for scalable and faithful
  explanations for LLMs' generated content will continue to grow. There have been
  major developments in the explainability of neural network models over the past
  decade. Among them, post-hoc explainability methods, especially Shapley values,
  have proven effective for interpreting deep learning models. However, there are
  major challenges in scaling up Shapley values for LLMs, particularly when
  dealing with long input contexts containing thousands of tokens and
  autoregressively generated output sequences. Furthermore, it is often unclear
  how to effectively utilize generated explanations to improve the performance of
  LLMs. In this paper, we introduce TextGenSHAP, an efficient post-hoc
  explanation method incorporating LM-specific techniques. We demonstrate that
  this leads to significant increases in speed compared to conventional Shapley
  value computations, reducing processing times from hours to minutes for
  token-level explanations, and to just seconds for document-level explanations.
  In addition, we demonstrate how real-time Shapley values can be utilized in two
  important scenarios, providing better understanding of long-document question
  answering by localizing important words and sentences; and improving existing
  document retrieval systems through enhancing the accuracy of selected passages
  and ultimately the final responses.
paper_ai_keywords: Shapley values post-hoc explainability methods neural network models black-boxes complex reasoning processes LM-specific identified token-level explanations document-level explanations localizing important words and sentences document retrieval systems