Large Language Models as Tax Attorneys: A Case Study in Legal
  Capabilities Emergence
Better understanding of Large Language Models' (LLMs) legal analysis
abilities can contribute to improving the efficiency of legal services,
governing artificial intelligence, and leveraging LLMs to identify
inconsistencies in law. This paper explores LLM capabilities in applying tax
law. We choose this area of law because it has a structure that allows us to
set up automated validation pipelines across thousands of examples, requires
logical reasoning and maths skills, and enables us to test LLM capabilities in
a manner relevant to real-world economic lives of citizens and companies. Our
experiments demonstrate emerging legal understanding capabilities, with
improved performance in each subsequent OpenAI model release. We experiment
with retrieving and utilising the relevant legal authority to assess the impact
of providing additional legal context to LLMs. Few-shot prompting, presenting
examples of question-answer pairs, is also found to significantly enhance the
performance of the most advanced model, GPT-4. The findings indicate that LLMs,
particularly when combined with prompting enhancements and the correct legal
texts, can perform at high levels of accuracy but not yet at expert tax lawyer
levels. As LLMs continue to advance, their ability to reason about law
autonomously could have significant implications for the legal profession and
AI governance.
6487bf34e1fae0ea14e30ca5
John J. Nay
admin_assigned
2023-06-13T15:01:54.316Z
63f2b49f7ddf724fbcc534fd
/avatars/4076a12ac9423ab276d06334c7e392db.svg
John Nay
user
johnjnay
6487bf34e1fae0ea14e30ca6
David Karamardian
6487bf34e1fae0ea14e30ca7
Sarah B. Lawsky
6487bf34e1fae0ea14e30ca8
Wenting Tao
6487bf34e1fae0ea14e30ca9
Meghana Bhat
6487bf34e1fae0ea14e30caa
Raghav Jain
6487bf34e1fae0ea14e30cab
Aaron Travis Lee
6487bf34e1fae0ea14e30cac
Jonathan H. Choi
6487bf34e1fae0ea14e30cad
Jungo Kasai
admin_assigned
2023-06-14T12:30:24.107Z
62908273c740ebb981a6dba4
/avatars/465f50369c367b07670f5209c83d65f2.svg
Jungo Kasai
user
jungok
Large Language Models as Tax Attorneys: A Case Study in Legal
  Capabilities Emergence
Better understanding of Large Language Models' (LLMs) legal analysis
abilities can contribute to improving the efficiency of legal services,
governing artificial intelligence, and leveraging LLMs to identify
inconsistencies in law. This paper explores LLM capabilities in applying tax
law. We choose this area of law because it has a structure that allows us to
set up automated validation pipelines across thousands of examples, requires
logical reasoning and maths skills, and enables us to test LLM capabilities in
a manner relevant to real-world economic lives of citizens and companies. Our
experiments demonstrate emerging legal understanding capabilities, with
improved performance in each subsequent OpenAI model release. We experiment
with retrieving and utilising the relevant legal authority to assess the impact
of providing additional legal context to LLMs. Few-shot prompting, presenting
examples of question-answer pairs, is also found to significantly enhance the
performance of the most advanced model, GPT-4. The findings indicate that LLMs,
particularly when combined with prompting enhancements and the correct legal
texts, can perform at high levels of accuracy but not yet at expert tax lawyer
levels. As LLMs continue to advance, their ability to reason about law
autonomously could have significant implications for the legal profession and
AI governance.
LLMs demonstrate increasing legal understanding and accuracy in tax law when provided with additional context and prompting enhancements, though they have not yet reached the level of expert tax lawyers.
Large Language Models LLMs tax law automated validation pipelines logical reasoning few-shot prompting GPT-4 AI governance