title: CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge
    Graphs in the LLM Era
summary: Retrieval from graph data is crucial for augmenting large language models
  (LLM) with both open-domain knowledge and private enterprise data, and it is
  also a key component in the recent GraphRAG system (edge et al., 2024). Despite
  decades of research on knowledge graphs and knowledge base question answering,
  leading LLM frameworks (e.g. Langchain and LlamaIndex) have only minimal
  support for retrieval from modern encyclopedic knowledge graphs like Wikidata.
  In this paper, we analyze the root cause and suggest that modern RDF knowledge
  graphs (e.g. Wikidata, Freebase) are less efficient for LLMs due to overly
  large schemas that far exceed the typical LLM context window, use of resource
  identifiers, overlapping relation types and lack of normalization. As a
  solution, we propose property graph views on top of the underlying RDF graph
  that can be efficiently queried by LLMs using Cypher. We instantiated this idea
  on Wikidata and introduced CypherBench, the first benchmark with 11
  large-scale, multi-domain property graphs with 7.8 million entities and over
  10,000 questions. To achieve this, we tackled several key challenges, including
  developing an RDF-to-property graph conversion engine, creating a systematic
  pipeline for text-to-Cypher task generation, and designing new evaluation
  metrics.
paper_authors._id: 6771fca3117cc54ff8b99a7f
paper_authors.name: Yanlin Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T20:03:01.909Z
paper_authors._id: 642f3e9ac953ca48ccd10927
paper_authors.avatarUrl: /avatars/eb6da3fce184aeb8cb9c2dfcd4235206.svg
paper_authors.fullname: Yanlin Feng
paper_authors.type: user
paper_authors.user: yanlinf
paper_authors._id: 6771fca3117cc54ff8b99a80
paper_authors.name: Simone Papicchio
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-31T15:09:48.747Z
paper_authors._id: 62e10decd3a32f2d445056e7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e10decd3a32f2d445056e7/RgbtYjNbTnuh1KrtZqZhI.png
paper_authors.fullname: Simone PAPICCHIO
paper_authors.type: user
paper_authors.user: simone-papicchio
paper_authors._id: 6771fca3117cc54ff8b99a81
paper_authors.name: Sajjadur Rahman
paper_title: CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge
    Graphs in the LLM Era
paper_summary: Retrieval from graph data is crucial for augmenting large language models
  (LLM) with both open-domain knowledge and private enterprise data, and it is
  also a key component in the recent GraphRAG system (edge et al., 2024). Despite
  decades of research on knowledge graphs and knowledge base question answering,
  leading LLM frameworks (e.g. Langchain and LlamaIndex) have only minimal
  support for retrieval from modern encyclopedic knowledge graphs like Wikidata.
  In this paper, we analyze the root cause and suggest that modern RDF knowledge
  graphs (e.g. Wikidata, Freebase) are less efficient for LLMs due to overly
  large schemas that far exceed the typical LLM context window, use of resource
  identifiers, overlapping relation types and lack of normalization. As a
  solution, we propose property graph views on top of the underlying RDF graph
  that can be efficiently queried by LLMs using Cypher. We instantiated this idea
  on Wikidata and introduced CypherBench, the first benchmark with 11
  large-scale, multi-domain property graphs with 7.8 million entities and over
  10,000 questions. To achieve this, we tackled several key challenges, including
  developing an RDF-to-property graph conversion engine, creating a systematic
  pipeline for text-to-Cypher task generation, and designing new evaluation
  metrics.
paper_ai_keywords: graph data large language models (LLM) GraphRAG system knowledge graphs knowledge base question answering denoising autoencoders RDF knowledge graphs Wikidata Freebase LLM context window resource identifiers relation types property graph views Cypher CypherBench RDF-to-property graph conversion engine text-to-Cypher task generation evaluation metrics