Target-Oriented Pretraining Data Selection via Neuron-Activated Graph
| Authors | Zijun Wang et al. |
| Year | 2026 |
| HF Upvotes | 10 |
| arXiv | 2604.15706 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Everyday tasks come with a target, and pretraining models around this target is what turns them into experts. In this paper, we study target-oriented language model (LM) pretraining by introducing Neuron-Activated Graph Ranking (NAG-based Ranking), a training-free and interpretable framework for target pretraining data selection. Rather than using black-box representations, our approach directly characterizes each target input by a sparse set of high-impact neurons in any off-the-shelf LLMs. Concretely, we quantify neuron impact and select the most influential neurons across layers into a compact Neuron-Activated Graph (NAG), and rank candidate data by NAG similarity to target examples. We conduct experiments across six benchmarks, where our NAG-based Ranking improves target-oriented pretraining by 4.9% on average over random sampling, and also outperforms state-of-the-art baselines by 5.3% accuracy on HellaSwag. It also remains effective under a more applicable multi-target setting, where our best setup surpasses two baselines by 1.1% and 4.1%, respectively. Furthermore, we provide a comprehensive analysis on why and how our NAG works, e.g., deactivating NAG-selected neurons (only 0.12% of all) causes a 23.5% performance collapse, and restricting NAG to the final layer incurs a 4.1% average drop, indicating that NAG captures a sparse "functional backbone" for learning target features. We release the code at https://github.com/asillycat/NAG.
Engineering Breakdown
Plain English
This paper tackles the problem of selecting pretraining data strategically when you want to turn a general language model into a specialist for a specific task or domain. Rather than training on random data or using expensive neural network representations, the authors propose NAG-based Ranking, which identifies the most important neurons in an existing LLM that respond to your target task, then uses those neurons to rank and select relevant pretraining data. The approach is training-free (you don't retrain the model to find important neurons) and interpretable (you can see which neurons matter), and it shows improvements across six benchmarks by focusing pretraining data selection on examples that activate the right neurons for your target.
Core Technical Contribution
The core innovation is Neuron-Activated Graph (NAG), a sparse, interpretable representation of what makes data relevant to a specific target task. Instead of relying on opaque embedding similarities or fine-tuning-based importance scores, the authors directly measure which neurons across all layers of an off-the-shelf LLM activate most strongly on target examples, then build a compact graph of these high-impact neurons. This graph becomes the signature of the target task, and candidate pretraining data is ranked by how similarly it activates this same neuron set. The training-free aspect is critical—you don't need to fine-tune or train anything to identify the neurons; you just run target examples through the model and measure activation patterns.
How It Works
The mechanism has three main phases. First, given a set of target examples (the task or domain you care about), you feed them through an off-the-shelf LLM and measure neuron activations across all layers. You identify the neurons with the highest impact on the target task—likely using a metric like gradient-based importance or activation magnitude—and construct a Neuron-Activated Graph that represents these critical neurons and their relationships. Second, for each candidate data point in your pretraining corpus, you compute its activation pattern across the same neurons and measure the similarity of that pattern to the target NAG (probably using cosine similarity or another graph similarity metric). Third, you rank all candidate data by this similarity score and select the top-k examples for target-oriented pretraining, which should now preferentially include data that activates the same neurons as your target task. The ranking is deterministic and requires only one forward pass per candidate, making it computationally tractable even on large corpora.
Production Impact
For teams building specialized language models (e.g., code generation models, medical QA systems, financial analysis tools), this approach replaces expensive heuristics and trial-and-error data curation with a principled, automated selection mechanism. Instead of manually labeling which pretraining data is relevant or running expensive fine-tuning experiments to estimate data importance, you can now generate rankings in hours using only inference. The production pipeline simplifies: (1) define your target task with a small set of representative examples, (2) run NAG once on your existing model to identify task-critical neurons, (3) score all candidate data against that NAG, (4) feed the top-ranked examples into pretraining. This reduces data pipeline complexity and makes it feasible to A/B test different target domains without retraining the ranking model. However, the approach adds latency if you have millions of candidate documents (one forward pass per document), and the effectiveness depends on having a good base LLM that generalizes the neuron importance patterns you observe.
Limitations and When Not to Use This
The paper assumes that neuron importance patterns in a base LLM transfer to the pretraining phase, but this may not hold if the base model and pretraining setup are very different (different architectures, scales, or training objectives). NAG-based Ranking is also inherently tied to a specific model—the neurons that matter for one LLM architecture may not transfer to another, so you'd need to recompute NAGs if you switch base models. The approach may struggle with targets that are very different from the base model's pretraining distribution (e.g., using a general English model to rank data for a low-resource language task). Additionally, the paper doesn't deeply address how to choose which neurons to include in the NAG or tune the ranking threshold; these design choices could significantly impact results but require some hyperparameter tuning or heuristics. Finally, interpretability claims are somewhat limited—while we can identify which neurons matter, we still have limited insight into why those neurons matter or what linguistic features they capture.
Research Context
This work builds on a growing body of research in neuron importance and model interpretability (e.g., lottery ticket hypothesis, neural network pruning) but applies it to the novel problem of target-oriented data selection. It extends prior work in curriculum learning and data selection that typically required fine-tuning or gradient-based importance scores (slower and model-dependent). The paper connects to recent interest in making pretraining more efficient and targeted, particularly relevant as models grow larger and pretraining data becomes a bottleneck. It also fits within the broader effort to make neural model behavior more interpretable and controllable, moving away from pure black-box similarity metrics toward sparse, human-understandable neuron patterns as model signatures.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::