title: Large Language Models as Tax Attorneys: A Case Study in Legal
    Capabilities Emergence
summary: Better understanding of Large Language Models' (LLMs) legal analysis
  abilities can contribute to improving the efficiency of legal services,
  governing artificial intelligence, and leveraging LLMs to identify
  inconsistencies in law. This paper explores LLM capabilities in applying tax
  law. We choose this area of law because it has a structure that allows us to
  set up automated validation pipelines across thousands of examples, requires
  logical reasoning and maths skills, and enables us to test LLM capabilities in
  a manner relevant to real-world economic lives of citizens and companies. Our
  experiments demonstrate emerging legal understanding capabilities, with
  improved performance in each subsequent OpenAI model release. We experiment
  with retrieving and utilising the relevant legal authority to assess the impact
  of providing additional legal context to LLMs. Few-shot prompting, presenting
  examples of question-answer pairs, is also found to significantly enhance the
  performance of the most advanced model, GPT-4. The findings indicate that LLMs,
  particularly when combined with prompting enhancements and the correct legal
  texts, can perform at high levels of accuracy but not yet at expert tax lawyer
  levels. As LLMs continue to advance, their ability to reason about law
  autonomously could have significant implications for the legal profession and
  AI governance.
paper_authors._id: 6487bf34e1fae0ea14e30ca5
paper_authors.name: John J. Nay
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-13T15:01:54.316Z
paper_authors._id: 63f2b49f7ddf724fbcc534fd
paper_authors.avatarUrl: /avatars/4076a12ac9423ab276d06334c7e392db.svg
paper_authors.fullname: John Nay
paper_authors.type: user
paper_authors.user: johnjnay
paper_authors._id: 6487bf34e1fae0ea14e30ca6
paper_authors.name: David Karamardian
paper_authors._id: 6487bf34e1fae0ea14e30ca7
paper_authors.name: Sarah B. Lawsky
paper_authors._id: 6487bf34e1fae0ea14e30ca8
paper_authors.name: Wenting Tao
paper_authors._id: 6487bf34e1fae0ea14e30ca9
paper_authors.name: Meghana Bhat
paper_authors._id: 6487bf34e1fae0ea14e30caa
paper_authors.name: Raghav Jain
paper_authors._id: 6487bf34e1fae0ea14e30cab
paper_authors.name: Aaron Travis Lee
paper_authors._id: 6487bf34e1fae0ea14e30cac
paper_authors.name: Jonathan H. Choi
paper_authors._id: 6487bf34e1fae0ea14e30cad
paper_authors.name: Jungo Kasai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T12:30:24.107Z
paper_authors._id: 62908273c740ebb981a6dba4
paper_authors.avatarUrl: /avatars/465f50369c367b07670f5209c83d65f2.svg
paper_authors.fullname: Jungo Kasai
paper_authors.type: user
paper_authors.user: jungok
paper_title: Large Language Models as Tax Attorneys: A Case Study in Legal
    Capabilities Emergence
paper_summary: Better understanding of Large Language Models' (LLMs) legal analysis
  abilities can contribute to improving the efficiency of legal services,
  governing artificial intelligence, and leveraging LLMs to identify
  inconsistencies in law. This paper explores LLM capabilities in applying tax
  law. We choose this area of law because it has a structure that allows us to
  set up automated validation pipelines across thousands of examples, requires
  logical reasoning and maths skills, and enables us to test LLM capabilities in
  a manner relevant to real-world economic lives of citizens and companies. Our
  experiments demonstrate emerging legal understanding capabilities, with
  improved performance in each subsequent OpenAI model release. We experiment
  with retrieving and utilising the relevant legal authority to assess the impact
  of providing additional legal context to LLMs. Few-shot prompting, presenting
  examples of question-answer pairs, is also found to significantly enhance the
  performance of the most advanced model, GPT-4. The findings indicate that LLMs,
  particularly when combined with prompting enhancements and the correct legal
  texts, can perform at high levels of accuracy but not yet at expert tax lawyer
  levels. As LLMs continue to advance, their ability to reason about law
  autonomously could have significant implications for the legal profession and
  AI governance.
paper_ai_keywords: Large Language Models (LLMs) legal analysis tax law automated validation pipelines logical reasoning maths skills emerging legal understanding capabilities OpenAI model release retrieving utilising relevant legal authority Few-shot prompting question-answer pairs GPT-4 expert tax lawyer levels reasoning about law autonomously