IPBench: Benchmarking the Knowledge of Large Language Models in
  Intellectual Property
Intellectual Property (IP) is a unique domain that integrates technical and
legal knowledge, making it inherently complex and knowledge-intensive. As large
language models (LLMs) continue to advance, they show great potential for
processing IP tasks, enabling more efficient analysis, understanding, and
generation of IP-related content. However, existing datasets and benchmarks
either focus narrowly on patents or cover limited aspects of the IP field,
lacking alignment with real-world scenarios. To bridge this gap, we introduce
the first comprehensive IP task taxonomy and a large, diverse bilingual
benchmark, IPBench, covering 8 IP mechanisms and 20 tasks. This benchmark is
designed to evaluate LLMs in real-world intellectual property applications,
encompassing both understanding and generation. We benchmark 16 LLMs, ranging
from general-purpose to domain-specific models, and find that even the
best-performing model achieves only 75.8% accuracy, revealing substantial room
for improvement. Notably, open-source IP and law-oriented models lag behind
closed-source general-purpose models. We publicly release all data and code of
IPBench and will continue to update it with additional IP-related tasks to
better reflect real-world challenges in the intellectual property domain.
68084b46fa5a6cc6bd9e6a83
Qiyao Wang
claimed_verified
2025-04-23T08:27:53.885Z
64560618bfdf9c63ce2d658a
https://cdn-avatars.huggingface.co/v1/production/uploads/64560618bfdf9c63ce2d658a/GVBWU4yNzRsjdyzKT3z3B.jpeg
Mathsion Wong
user
QiYao-Wang
68084b46fa5a6cc6bd9e6a84
Guhong Chen
68084b46fa5a6cc6bd9e6a85
Hongbo Wang
68084b46fa5a6cc6bd9e6a86
Huaren Liu
68084b46fa5a6cc6bd9e6a87
Minghui Zhu
68084b46fa5a6cc6bd9e6a88
Zhifei Qin
68084b46fa5a6cc6bd9e6a89
Linwei Li
68084b46fa5a6cc6bd9e6a8a
Yilin Yue
68084b46fa5a6cc6bd9e6a8b
Shiqiang Wang
68084b46fa5a6cc6bd9e6a8c
Jiayan Li
68084b46fa5a6cc6bd9e6a8d
Yihang Wu
68084b46fa5a6cc6bd9e6a8e
Ziqiang Liu
68084b46fa5a6cc6bd9e6a8f
Longze Chen
68084b46fa5a6cc6bd9e6a90
Run Luo
68084b46fa5a6cc6bd9e6a91
Liyang Fan
68084b46fa5a6cc6bd9e6a92
Jiaming Li
68084b46fa5a6cc6bd9e6a93
Lei Zhang
68084b46fa5a6cc6bd9e6a94
Kan Xu
68084b46fa5a6cc6bd9e6a95
Hongfei Lin
68084b46fa5a6cc6bd9e6a96
Hamid Alinejad-Rokny
68084b46fa5a6cc6bd9e6a97
Shiwen Ni
68084b46fa5a6cc6bd9e6a98
Yuan Lin
68084b46fa5a6cc6bd9e6a99
Min Yang
IPBench: Benchmarking the Knowledge of Large Language Models in
  Intellectual Property
Intellectual Property (IP) is a unique domain that integrates technical and
legal knowledge, making it inherently complex and knowledge-intensive. As large
language models (LLMs) continue to advance, they show great potential for
processing IP tasks, enabling more efficient analysis, understanding, and
generation of IP-related content. However, existing datasets and benchmarks
either focus narrowly on patents or cover limited aspects of the IP field,
lacking alignment with real-world scenarios. To bridge this gap, we introduce
the first comprehensive IP task taxonomy and a large, diverse bilingual
benchmark, IPBench, covering 8 IP mechanisms and 20 tasks. This benchmark is
designed to evaluate LLMs in real-world intellectual property applications,
encompassing both understanding and generation. We benchmark 16 LLMs, ranging
from general-purpose to domain-specific models, and find that even the
best-performing model achieves only 75.8% accuracy, revealing substantial room
for improvement. Notably, open-source IP and law-oriented models lag behind
closed-source general-purpose models. We publicly release all data and code of
IPBench and will continue to update it with additional IP-related tasks to
better reflect real-world challenges in the intellectual property domain.
A large, diverse bilingual benchmark, IPBench, is introduced to evaluate LLMs in real-world intellectual property applications, revealing substantial room for improvement.
LLMs large language models IP task taxonomy IPBench understanding generation domain-specific models