Mitigating Copy Bias in In-Context Learning through Neuron Pruning.
| Authors | Ameen Ali et al. |
| Year | 2026 |
| Venue | EACL 2026 |
| Paper | View on ACL Anthology |
| PDF | Download |
Abstract
Abstract not yet available in this stub. Read the full paper →
Engineering Breakdown
Plain English
This paper addresses 'copy bias' in in-context learning—where language models simply reproduce examples from the prompt instead of generalizing to solve new problems. The authors show that neuron pruning (removing unnecessary model parameters) can reduce this copy bias while maintaining performance, suggesting that the circuits responsible for copying can be surgically removed without harming the model's ability to actually learn from context.
Key Engineering Insight
Pruning specific neurons reduces copy bias because the model's copying behavior and genuine in-context learning are localized to different sets of neurons—you can remove one without destroying the other. This tells us that these are separable capabilities at the circuit level, not intertwined emergent properties.
Why It Matters for Engineers
Copy bias is a real failure mode in production systems using in-context learning for tasks like code generation, data extraction, and few-shot classification. If you can reliably remove this pathological behavior through pruning without retraining, it's a lightweight technique for fixing model behavior post-hoc, which matters when you can't afford full retraining or fine-tuning.
Research Context
In-context learning is known to suffer from shortcut solutions where models memorize and replay examples rather than generalize. Prior work identified copy bias as a problem but mostly addressed it through training changes. This work moves past that by showing the bias can be surgically fixed through structured pruning, opening a new angle on mechanistic interpretability—understanding that different behaviors live in different parts of the network.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::