title: IPBench: Benchmarking the Knowledge of Large Language Models in
    Intellectual Property
summary: Intellectual Property (IP) is a unique domain that integrates technical and
  legal knowledge, making it inherently complex and knowledge-intensive. As large
  language models (LLMs) continue to advance, they show great potential for
  processing IP tasks, enabling more efficient analysis, understanding, and
  generation of IP-related content. However, existing datasets and benchmarks
  either focus narrowly on patents or cover limited aspects of the IP field,
  lacking alignment with real-world scenarios. To bridge this gap, we introduce
  the first comprehensive IP task taxonomy and a large, diverse bilingual
  benchmark, IPBench, covering 8 IP mechanisms and 20 tasks. This benchmark is
  designed to evaluate LLMs in real-world intellectual property applications,
  encompassing both understanding and generation. We benchmark 16 LLMs, ranging
  from general-purpose to domain-specific models, and find that even the
  best-performing model achieves only 75.8% accuracy, revealing substantial room
  for improvement. Notably, open-source IP and law-oriented models lag behind
  closed-source general-purpose models. We publicly release all data and code of
  IPBench and will continue to update it with additional IP-related tasks to
  better reflect real-world challenges in the intellectual property domain.
paper_authors._id: 68084b46fa5a6cc6bd9e6a83
paper_authors.name: Qiyao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:27:53.885Z
paper_authors._id: 64560618bfdf9c63ce2d658a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64560618bfdf9c63ce2d658a/GVBWU4yNzRsjdyzKT3z3B.jpeg
paper_authors.fullname: Mathsion Wong
paper_authors.type: user
paper_authors.user: QiYao-Wang
paper_authors._id: 68084b46fa5a6cc6bd9e6a84
paper_authors.name: Guhong Chen
paper_authors._id: 68084b46fa5a6cc6bd9e6a85
paper_authors.name: Hongbo Wang
paper_authors._id: 68084b46fa5a6cc6bd9e6a86
paper_authors.name: Huaren Liu
paper_authors._id: 68084b46fa5a6cc6bd9e6a87
paper_authors.name: Minghui Zhu
paper_authors._id: 68084b46fa5a6cc6bd9e6a88
paper_authors.name: Zhifei Qin
paper_authors._id: 68084b46fa5a6cc6bd9e6a89
paper_authors.name: Linwei Li
paper_authors._id: 68084b46fa5a6cc6bd9e6a8a
paper_authors.name: Yilin Yue
paper_authors._id: 68084b46fa5a6cc6bd9e6a8b
paper_authors.name: Shiqiang Wang
paper_authors._id: 68084b46fa5a6cc6bd9e6a8c
paper_authors.name: Jiayan Li
paper_authors._id: 68084b46fa5a6cc6bd9e6a8d
paper_authors.name: Yihang Wu
paper_authors._id: 68084b46fa5a6cc6bd9e6a8e
paper_authors.name: Ziqiang Liu
paper_authors._id: 68084b46fa5a6cc6bd9e6a8f
paper_authors.name: Longze Chen
paper_authors._id: 68084b46fa5a6cc6bd9e6a90
paper_authors.name: Run Luo
paper_authors._id: 68084b46fa5a6cc6bd9e6a91
paper_authors.name: Liyang Fan
paper_authors._id: 68084b46fa5a6cc6bd9e6a92
paper_authors.name: Jiaming Li
paper_authors._id: 68084b46fa5a6cc6bd9e6a93
paper_authors.name: Lei Zhang
paper_authors._id: 68084b46fa5a6cc6bd9e6a94
paper_authors.name: Kan Xu
paper_authors._id: 68084b46fa5a6cc6bd9e6a95
paper_authors.name: Hongfei Lin
paper_authors._id: 68084b46fa5a6cc6bd9e6a96
paper_authors.name: Hamid Alinejad-Rokny
paper_authors._id: 68084b46fa5a6cc6bd9e6a97
paper_authors.name: Shiwen Ni
paper_authors._id: 68084b46fa5a6cc6bd9e6a98
paper_authors.name: Yuan Lin
paper_authors._id: 68084b46fa5a6cc6bd9e6a99
paper_authors.name: Min Yang
paper_title: IPBench: Benchmarking the Knowledge of Large Language Models in
    Intellectual Property
paper_summary: Intellectual Property (IP) is a unique domain that integrates technical and
  legal knowledge, making it inherently complex and knowledge-intensive. As large
  language models (LLMs) continue to advance, they show great potential for
  processing IP tasks, enabling more efficient analysis, understanding, and
  generation of IP-related content. However, existing datasets and benchmarks
  either focus narrowly on patents or cover limited aspects of the IP field,
  lacking alignment with real-world scenarios. To bridge this gap, we introduce
  the first comprehensive IP task taxonomy and a large, diverse bilingual
  benchmark, IPBench, covering 8 IP mechanisms and 20 tasks. This benchmark is
  designed to evaluate LLMs in real-world intellectual property applications,
  encompassing both understanding and generation. We benchmark 16 LLMs, ranging
  from general-purpose to domain-specific models, and find that even the
  best-performing model achieves only 75.8% accuracy, revealing substantial room
  for improvement. Notably, open-source IP and law-oriented models lag behind
  closed-source general-purpose models. We publicly release all data and code of
  IPBench and will continue to update it with additional IP-related tasks to
  better reflect real-world challenges in the intellectual property domain.
paper_ai_summary: A large, diverse bilingual benchmark, IPBench, is introduced to evaluate LLMs in real-world intellectual property applications, revealing substantial room for improvement.
paper_ai_keywords: LLMs large language models IP task taxonomy IPBench understanding generation domain-specific models