Tracing the Roots: A Multi-Agent Framework for Uncovering Data Lineage in Post-Training LLMs
| Authors | Yu Li et al. |
| Year | 2026 |
| HF Upvotes | 20 |
| arXiv | 2604.10480 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Post-training data plays a pivotal role in shaping the capabilities of Large Language Models (LLMs), yet datasets are often treated as isolated artifacts, overlooking the systemic connections that underlie their evolution. To disentangle these complex relationships, we introduce the concept of data lineage to the LLM ecosystem and propose an automated multi-agent framework to reconstruct the evolutionary graph of dataset development. Through large-scale lineage analysis, we characterize domain-specific structural patterns, such as vertical refinement in math-oriented datasets and horizontal aggregation in general-domain corpora. Moreover, we uncover pervasive systemic issues, including structural redundancy induced by implicit dataset intersections and the propagation of benchmark contamination along lineage paths. To demonstrate the practical value of lineage analysis for data construction, we leverage the reconstructed lineage graph to create a lineage-aware diversity-oriented dataset. By anchoring instruction sampling at upstream root sources, this approach mitigates downstream homogenization and hidden redundancy, yielding a more diverse post-training corpus. We further highlight lineage-centric analysis as an efficient and robust topological alternative to sample-level dataset comparison for large-scale data ecosystems. By grounding data construction in explicit lineage structures, our work advances post-training data curation toward a more systematic and controllable paradigm.
Engineering Breakdown
Plain English
This paper introduces data lineage tracking to understand how datasets evolve and interconnect in the LLM ecosystem. The authors built an automated multi-agent framework that reconstructs the evolutionary history of datasets used in post-training, revealing patterns like vertical refinement (math datasets becoming progressively specialized) and horizontal aggregation (general-domain corpora combining multiple sources). The analysis uncovered critical systemic issues: datasets often contain hidden overlaps that create structural redundancy, and benchmark contamination propagates through the lineage chain. This work treats datasets not as isolated artifacts but as connected systems with traceable ancestry and dependencies.
Core Technical Contribution
The core novelty is applying data lineage—a concept from data engineering and databases—to the LLM training ecosystem for the first time. The authors developed an automated multi-agent framework that can reconstruct how datasets were derived from or combined with other datasets, creating an evolutionary graph of dataset development at scale. This enables characterization of domain-specific structural patterns (vertical vs. horizontal evolution) that were previously invisible. The key insight is that understanding dataset relationships systematically surfaces pervasive issues like implicit intersections and contamination propagation that traditional dataset evaluation methods miss.
How It Works
The framework operates as a multi-agent system where agents analyze dataset metadata, documentation, and provenance information to infer relationships between datasets. Starting with a collection of post-training datasets (the input), agents identify explicit dependencies (dataset X was derived from dataset Y) and implicit connections (dataset X and Y share common source material). These relationships are encoded into a directed acyclic graph representing the evolutionary history. The system then analyzes this graph to identify structural patterns: vertical refinement occurs when datasets are progressively filtered/curated along a single lineage chain, while horizontal aggregation happens when multiple independent datasets are merged. Finally, the framework traces contamination pathways—if a dataset appears in multiple lineages, that contamination signal propagates to all downstream datasets in the graph. Output is a comprehensive lineage visualization plus quantified metrics on redundancy and contamination risk.
Production Impact
For engineers building LLM training pipelines, this work provides actionable intelligence for data curation and quality assurance. You could implement automated lineage tracking to detect hidden dataset overlaps before training—preventing the expensive mistake of training on redundant data that wastes compute. For benchmark and evaluation teams, lineage analysis reveals contamination risks upstream: if your test set lineage intersects with training data lineages, you know model performance is artificially inflated before spending resources on evaluation. The trade-off is operational: maintaining lineage metadata requires disciplined dataset versioning and documentation (similar to software supply chain practices), and the multi-agent analysis adds upfront analysis cost. However, the ROI is substantial—catching redundancy early prevents wasted training runs, and contamination detection prevents downstream models from being deployed with inflated benchmark numbers.
Limitations and When Not to Use This
The framework assumes reliable access to dataset provenance metadata and documentation, which may be incomplete or missing for datasets from non-academic sources or legacy systems. The multi-agent approach's accuracy depends on its ability to infer implicit dataset relationships from imperfect metadata—it may miss contamination when datasets share content without explicit documentation or falsely flag unrelated datasets with similar statistical properties as related. The paper focuses on structural analysis of post-training datasets but doesn't address contamination in pretraining data, which operates at vastly larger scale and different provenance patterns. Additionally, the approach may not scale effectively to massive numbers of proprietary or internal datasets where lineage information is intentionally obscured, limiting its applicability in large industry settings.
Research Context
This work builds on two established research directions: (1) dataset quality and benchmark contamination concerns highlighted in recent critiques of LLM evaluation practices, and (2) data governance and lineage tracking from data engineering and ML Ops communities. It connects to prior work on dataset bias, synthetic data generation, and the growing recognition that LLM capabilities are fundamentally bounded by post-training data quality. The paper opens a new research direction in treating datasets as first-class artifacts with traceable dependencies—similar to how software engineering evolved version control and dependency management for code. This framework could enable new research into optimal dataset composition, hereditary contamination patterns, and principled approaches to data curation that go beyond current ad-hoc practices.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::