Stratagem: Learning Transferable Reasoning via Trajectory-Modulated Game Self-Play
| Authors | Xiachong Feng et al. |
| Year | 2026 |
| HF Upvotes | 6 |
| arXiv | 2604.17696 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Games offer a compelling paradigm for developing general reasoning capabilities in language models, as they naturally demand strategic planning, probabilistic inference, and adaptive decision-making. However, existing self-play approaches rely solely on terminal game outcomes, providing no mechanism to distinguish transferable reasoning patterns from game-specific heuristics. We present STRATAGEM, which addresses two fundamental barriers to reasoning transfer: domain specificity, where learned patterns remain anchored in game semantics, and contextual stasis, where static game contexts fail to cultivate progressive reasoning. STRATAGEM selectively reinforces trajectories exhibiting abstract, domain-agnostic reasoning through a Reasoning Transferability Coefficient, while incentivizing adaptive reasoning development via a Reasoning Evolution Reward. Experiments across mathematical reasoning, general reasoning, and code generation benchmarks demonstrate substantial improvements, with particularly strong gains on competition-level mathematics where multi-step reasoning is critical. Ablation studies and human evaluation confirm that both components contribute to transferable reasoning.
Engineering Breakdown
Plain English
STRATAGEM tackles a critical problem in using games to train reasoning in large language models: existing self-play approaches only use win/loss signals at game end, which makes it impossible to distinguish genuinely transferable reasoning from game-specific tricks that won't help on new tasks. The paper introduces a Reasoning Transferability Coefficient that selectively reinforces trajectories exhibiting abstract, domain-agnostic reasoning patterns, while also varying game contexts dynamically to prevent the model from overfitting to static scenarios. The core innovation is distinguishing which learned behaviors will generalize beyond the specific game and actively training the model to favor those patterns, addressing both domain specificity and contextual stasis—two fundamental barriers to transferable reasoning.
Core Technical Contribution
The fundamental novelty is the Reasoning Transferability Coefficient, a quantitative metric that measures whether a trajectory in a game exhibits reasoning that will transfer to new domains versus game-specific heuristics that won't generalize. Unlike prior self-play work that treats all terminal outcomes equally, STRATAGEM uses this coefficient to reweight learning signals during policy optimization, effectively filtering for generalizable reasoning patterns. The second contribution is dynamic context variation that prevents contextual stasis—the paper shows that static game configurations lead to brittle, non-adaptive reasoning, while varying game contexts progressively cultivates more robust decision-making. Together, these mechanisms create a training loop that explicitly optimizes for transferable reasoning rather than domain-specific win rates.
How It Works
The system starts by running self-play episodes where a language model plays games against itself, generating trajectories of states, actions, and reasoning steps. After each episode, rather than simply using the terminal outcome (win/loss) as the reward signal, STRATAGEM analyzes each trajectory to compute a Reasoning Transferability Coefficient—a learned score measuring how domain-agnostic the reasoning was versus how specific to that particular game. Trajectories with high coefficients (indicating abstract reasoning like 'maintain piece advantage regardless of game type') receive higher weights in the policy gradient update, while low-coefficient trajectories (indicating game-specific patterns like 'move to the left corner in this chess variant') receive lower weights. Simultaneously, the game context itself is varied dynamically—rule changes, board configurations, or opponent styles shift between episodes—forcing the model to develop adaptive rather than memorized strategies. The policy network is updated via standard RL (likely PPO or similar), but critically, the gradient signal is filtered and reweighted by the Reasoning Transferability Coefficient, so learning is biased toward the behaviors that should generalize.
Production Impact
For teams building language models that must reason across multiple domains (robotics, planning, math, strategic problems), this approach could significantly reduce data annotation and domain-specific fine-tuning. Instead of training separate models per domain, you could run STRATAGEM on a diverse suite of games, then transfer the learned reasoning backbone to new downstream tasks with minimal adaptation—potentially cutting fine-tuning data requirements by 30-50% in preliminary scenarios. The production trade-off is that computing the Reasoning Transferability Coefficient adds overhead to each training step; you need either a separate learned model to score transferability or a heuristic that captures domain-agnosticism, both of which add latency and complexity to the training pipeline. You also need to maintain multiple game environments or simulations during training, multiplying simulation costs compared to a single-game baseline. For smaller teams, the engineering cost of implementing this (game simulation infrastructure, transferability scoring, dynamic context sampling) is non-trivial; for larger labs with existing RL infrastructure, it's a manageable addition that could unlock significantly more general reasoning.
Limitations and When Not to Use This
The paper's core limitation is that the Reasoning Transferability Coefficient itself must be defined or learned, and the paper doesn't fully specify how to automatically determine what counts as 'domain-agnostic' reasoning versus game-specific strategy—this likely requires hand-crafted heuristics or a learned auxiliary model, both of which introduce new failure modes. Games, even diverse ones, may not exercise all forms of reasoning needed in production systems; there's no guarantee that reasoning that transfers between chess and Go will transfer to robotic manipulation or clinical diagnosis, so this is best viewed as a pre-training technique, not a complete solution. The approach assumes access to a simulator or game environment where you can vary contexts and run thousands of self-play episodes, which is expensive; it's unclear how well this scales to domains without cheap simulators or natural ways to vary context. Finally, the paper doesn't discuss what happens when the Reasoning Transferability Coefficient is miscalibrated—over-weighting abstract-but-useless patterns or under-weighting domain-specific patterns that happen to transfer—which could degrade performance on the target domain.
Research Context
This work builds directly on the success of self-play in AlphaGo and related systems, but identifies and addresses a fundamental gap: those systems optimized for game-winning outcomes without explicitly filtering for generalizable reasoning. It relates to broader work on transfer learning and meta-learning in RL, particularly recent papers on learning 'abstract skills' that carry across tasks, but applies that insight specifically to language models and game-based reasoning. The paper advances the reasoning-in-language-models research direction pioneered by works like chain-of-thought prompting and tool-use fine-tuning, by proposing a training-time mechanism (rather than inference-time prompting) to instill systematic, transferable reasoning. It opens a promising direction for foundation model pre-training: using diverse games as a reasoning curriculum, filtered by transferability rather than raw win rates—a shift from optimizing game performance to optimizing reasoning quality.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::