title: HiddenTables & PyQTax: A Cooperative Game and Dataset For TableQA to
    Ensure Scale and Data Privacy Across a Myriad of Taxonomies
summary: A myriad of different Large Language Models (LLMs) face a common challenge in
  contextually analyzing table question-answering tasks. These challenges are
  engendered from (1) finite context windows for large tables, (2) multi-faceted
  discrepancies amongst tokenization patterns against cell boundaries, and (3)
  various limitations stemming from data confidentiality in the process of using
  external models such as gpt-3.5-turbo. We propose a cooperative game dubbed
  "HiddenTables" as a potential resolution to this challenge. In essence,
  "HiddenTables" is played between the code-generating LLM "Solver" and the
  "Oracle" which evaluates the ability of the LLM agents to solve Table QA tasks.
  This game is based on natural language schemas and importantly, ensures the
  security of the underlying data. We provide evidential experiments on a diverse
  set of tables that demonstrate an LLM's collective inability to generalize and
  perform on complex queries, handle compositional dependencies, and align
  natural language to programmatic commands when concrete table schemas are
  provided. Unlike encoder-based models, we have pushed the boundaries of
  "HiddenTables" to not be limited by the number of rows - therefore we exhibit
  improved efficiency in prompt and completion tokens. Our infrastructure has
  spawned a new dataset "PyQTax" that spans across 116,671 question-table-answer
  triplets and provides additional fine-grained breakdowns & labels for varying
  question taxonomies. Therefore, in tandem with our academic contributions
  regarding LLMs' deficiency in TableQA tasks, "HiddenTables" is a tactile
  manifestation of how LLMs can interact with massive datasets while ensuring
  data security and minimizing generation costs.
paper_authors._id: 66719232e6252998d9db172e
paper_authors.name: William Watson
paper_authors._id: 66719232e6252998d9db172f
paper_authors.name: Nicole Cho
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-24T16:27:44.074Z
paper_authors._id: 6630a380b6d15534202ab83f
paper_authors.avatarUrl: /avatars/69af9465544b668c65ad4d9866fbfd06.svg
paper_authors.fullname: Nicole Cho
paper_authors.type: user
paper_authors.user: NicoleCho
paper_authors._id: 66719232e6252998d9db1730
paper_authors.name: Tucker Balch
paper_authors._id: 66719232e6252998d9db1731
paper_authors.name: Manuela Veloso
paper_title: HiddenTables & PyQTax: A Cooperative Game and Dataset For TableQA to
    Ensure Scale and Data Privacy Across a Myriad of Taxonomies
paper_summary: A myriad of different Large Language Models (LLMs) face a common challenge in
  contextually analyzing table question-answering tasks. These challenges are
  engendered from (1) finite context windows for large tables, (2) multi-faceted
  discrepancies amongst tokenization patterns against cell boundaries, and (3)
  various limitations stemming from data confidentiality in the process of using
  external models such as gpt-3.5-turbo. We propose a cooperative game dubbed
  "HiddenTables" as a potential resolution to this challenge. In essence,
  "HiddenTables" is played between the code-generating LLM "Solver" and the
  "Oracle" which evaluates the ability of the LLM agents to solve Table QA tasks.
  This game is based on natural language schemas and importantly, ensures the
  security of the underlying data. We provide evidential experiments on a diverse
  set of tables that demonstrate an LLM's collective inability to generalize and
  perform on complex queries, handle compositional dependencies, and align
  natural language to programmatic commands when concrete table schemas are
  provided. Unlike encoder-based models, we have pushed the boundaries of
  "HiddenTables" to not be limited by the number of rows - therefore we exhibit
  improved efficiency in prompt and completion tokens. Our infrastructure has
  spawned a new dataset "PyQTax" that spans across 116,671 question-table-answer
  triplets and provides additional fine-grained breakdowns & labels for varying
  question taxonomies. Therefore, in tandem with our academic contributions
  regarding LLMs' deficiency in TableQA tasks, "HiddenTables" is a tactile
  manifestation of how LLMs can interact with massive datasets while ensuring
  data security and minimizing generation costs.
paper_ai_keywords: Large Language Models (LLMs) contextually analyzing table question-answering tasks context windows tokenization patterns cell boundaries HiddenTables code-generating LLM Oracle natural language schemas encoder-based models prompt and completion tokens PyQTax question-table-answer triplets fine-grained breakdowns labels question taxonomies TableQA tasks