GraphRAG Best Practices: A Modular Evaluation Framework and Regulatory Case Study

Studies on the semantic web

Abstract

Graph Retrieval-Augmented Generation (GraphRAG) enhances Large Language Models by integrating knowledge graphs into retrieval and generation. Despite growing adoption, no systematic study has isolated the contribution of individual pipeline components to end-to-end performance, leaving practitioners with-out evidence-based configuration guidance. We address this gap with a component-level empirical study spanning three GraphRAG stages, namely graph-based indexing, graph-guided retrieval, and graph-enhanced generation, and identify which design decisions most significantly influence answer quality. Across three datasets (MSMARCO, HotpotQA, and a domain-specific EU banking regulation corpus), we benchmark 17 embedding models, six retriever types with varying traversal depths, nine reranking models, and four graph serialisation formats for prompt construction. Our analysis reveals three key findings: (1) retriever combination strategy and traversal depth are dataset-dependent, with domain-specific corpora favouring shallow traversal and separate reranking; (2) a dual-retriever pairing hypothetical-question embeddings with entity-based retrieval, pruned by a cross-encoder reranker, yields the most robust performance across domains (F1 up to 89.9); and (3) code-like graph serialisation (GraphML) provides the best quality– latency trade-off for generation, likely due to LLM familiarity with structured syntax. Applied to 500 questions over EU banking regulations (CRR + CRD IV), the optimised GraphRAG pipeline achieves win rates of 58–67% against Naive, HyDE, and Hybrid RAG baselines across comprehensiveness, diversity, empowerment, and correctness. All generation components use open-source LLMs (Llama-3.3-70B-Instruct), and datasets are publicly available, supporting reproducibility.

Authors 7

1. Yan-Ying Liao Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University, Aachen, Germany

2. Diego Collarana

   Fraunhofer Institute for Applied Information Technology · Universidad Privada Boliviana

   Affiliation as printed

   Fraunhofer FIT, Sankt Augustin, Germany

   Universidad Privada Boliviana, Cochabamba, Bolivia

3. Christopher Ingo Pack

   Fraunhofer Institute for Applied Information Technology

   Affiliation as printed

   Fraunhofer FIT, Sankt Augustin, Germany

4. Alexander Graß Aachen

   Fraunhofer Institute for Applied Information Technology · RWTH Aachen University

   Affiliation as printed

   Fraunhofer FIT, Sankt Augustin, Germany

   RWTH Aachen University, Aachen, Germany

5. Andreas Both

   Leipzig University of Applied Sciences

   Affiliation as printed

   Leipzig University of Applied Sciences, Leipzig, Germany

6. Stefan Decker Aachen

   Fraunhofer Institute for Applied Information Technology · RWTH Aachen University

   Affiliation as printed

   Fraunhofer FIT, Sankt Augustin, Germany

   RWTH Aachen University, Aachen, Germany

7. Christian Beecks

   FernUniversität in Hagen · Fraunhofer Institute for Applied Information Technology

   Affiliation as printed

   FernUniversität in Hagen, Hagen, Germany

   Fraunhofer FIT, Sankt Augustin, Germany