Hierarchical Abstract Tree for Cross-Document Retrieval-Augmented Generation
| Authors | Ziwen Zhao & Menglin Yang |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2605.00529 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Retrieval-augmented generation (RAG) enhances large language models with external knowledge, and tree-based RAG organizes documents into hierarchical indexes to support queries at multiple granularities. However, existing Tree-RAG methods designed for single-document retrieval face critical challenges in scaling to cross-document multi-hop questions: (1) poor distribution adaptability, where k-means clustering introduces noise due to rigid distribution assumptions; (2) structural isolation, as tree indexes lack explicit cross-document connections; and (3) coarse abstraction, which obscures fine-grained details. To address these limitations, we propose Ψ-RAG, a tree-RAG framework with two key components. First, a hierarchical abstract tree index built through an iterative "merging and collapse" process that adapts to data distributions without a priori assumption. Second, a multi-granular retrieval agent that intelligently interacts with the knowledge base with reorganized queries and an agent-powered hybrid retriever. Ψ-RAG supports diverse tasks from token-level question answering to document-level summarization. On cross-document multi-hop QA benchmarks, it outperforms RAPTOR by 25.9% and HippoRAG 2 by 7.4% in average F1 score. Code is available at https://github.com/Newiz430/Psi-RAG.
Engineering Breakdown
Plain English
This paper addresses a critical scaling problem in retrieval-augmented generation (RAG) systems: existing tree-based methods work well for single-document queries but fail when answering complex questions that require connecting information across multiple documents. The authors identify three specific failure modes: k-means clustering introduces noise due to rigid assumptions about data distribution, tree indexes don't explicitly connect information across documents, and the abstraction layers lose important fine-grained details. They propose Ψ-RAG, a new tree-RAG framework that builds hierarchical abstract trees through iterative merging and collapsing operations, enabling systems to answer cross-document multi-hop questions more effectively while maintaining both breadth and depth of information.
Core Technical Contribution
The core innovation is replacing rigid k-means-based clustering with an iterative merging-and-collapsing mechanism that adapts to actual document distribution rather than forcing data into predetermined clusters. The framework explicitly models cross-document connections at multiple hierarchical levels, allowing a single tree structure to answer both single-document and multi-document queries without architectural compromise. Rather than treating abstraction as a lossy process that discards details, Ψ-RAG preserves fine-grained information alongside coarse abstractions, enabling the retriever to trade off specificity and recall based on query complexity. This is fundamentally different from prior Tree-RAG methods which optimize exclusively for single-document retrieval and lack the cross-document linking capability.
How It Works
The system begins with raw documents and constructs a hierarchical abstract tree through an iterative bottom-up process: at each level, semantically similar chunks are merged together, while important granular details are preserved in a collapsing operation that creates bridges between merged nodes. Cross-document connections are explicitly maintained throughout the hierarchy—nodes at each level store references to related information in other documents, creating a graph-like structure overlaid on the tree. During retrieval, the system performs multi-hop traversal: it starts with a query, identifies relevant entry points across multiple documents in the tree, then traverses both within and across document boundaries to gather information needed for multi-hop reasoning. The ranking mechanism considers both the relevance score of nodes and the path cost of reaching them, allowing the system to balance direct relevance against information connectivity. At generation time, the LLM receives not just the retrieved chunks but also the hierarchical context showing how those chunks relate to each other within and across documents.
Production Impact
This approach directly solves the 'lost context problem' in cross-document QA systems—instead of independently retrieving snippets from different documents, Ψ-RAG retrieves them within their full hierarchical and cross-document context, reducing hallucination and improving reasoning quality on complex queries. Production systems using this would see measurable improvements on multi-hop benchmarks (e.g., HotpotQA-style questions requiring 2+ document hops), at the cost of higher indexing complexity: building and maintaining the hierarchical abstract tree requires additional preprocessing time and storage compared to flat retrieval. Integration requires careful tuning of the merging/collapsing thresholds to balance abstraction levels—too aggressive merging loses fine details, too conservative merging defeats the purpose of hierarchical structure. Latency impact is moderate: multi-hop traversal adds computational overhead compared to simple BM25 retrieval, but the hierarchical pruning reduces the search space, potentially offsetting gains from BM25 depending on document corpus size and query complexity distribution.
Limitations and When Not to Use This
The paper's abstract cuts off before describing the full technical approach, so complete evaluation metrics and scaling characteristics remain unclear—we don't know how performance degrades with corpus size (100K vs 10M documents) or what computational overhead the iterative tree-building process introduces. The method assumes documents have sufficient semantic structure for meaningful merging; it may perform poorly on highly heterogeneous collections or documents with little semantic overlap, where rigid clustering actually provides better isolation. The framework still requires tuning key hyperparameters (merge thresholds, abstraction levels, cross-document connection density) per domain, reducing its plug-and-play applicability. The explicit preservation of fine-grained details alongside abstractions increases memory requirements and may not scale efficiently to very large document collections without additional compression or pruning strategies.
Research Context
This work builds directly on the Tree-RAG research direction, which itself extends dense retrieval and hierarchical indexing work from classical IR (e.g., pyramid methods, summarization hierarchies). It addresses concrete failure modes identified in recent multi-hop QA research showing that flat retrieval methods and single-document tree methods both fail on benchmarks like HotpotQA and 2WikiMultihop. The paper contributes to the broader trend of moving beyond naive 'retrieve then generate' toward 'retrieve with structured context understanding'—similar to concurrent work in graph-augmented RAG and document-level reasoning. It opens a research direction on distribution-adaptive indexing that could influence how other hierarchical retrieval systems (for image search, code retrieval, etc.) handle multi-source queries.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::