How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data
| Authors | Zixian Huang et al. |
| Year | 2026 |
| HF Upvotes | 27 |
| arXiv | 2604.14164 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
A widely adopted strategy for model enhancement is to use synthetic data generated by a stronger model for supervised fine-tuning (SFT). However, for emerging reasoning models like Qwen3-8B, this approach often fails to improve reasoning capabilities and can even lead to a substantial drop in performance. In this work, we identify substantial stylistic divergence between teacher generated data and the distribution of student as a major factor impacting SFT. To bridge this gap, we propose a Teacher-Student Cooperation Data Synthesis framework (TESSY), which interleaves teacher and student models to alternately generate style and non-style tokens. Consequently, TESSY produces synthetic sequences that inherit the advanced reasoning capabilities of the teacher while maintaining stylistic consistency with the distribution of the student. In experiments on code generation using GPT-OSS-120B as the teacher, fine-tuning Qwen3-8B on teacher-generated data leads to performance drops of 3.25% on LiveCodeBench-Pro and 10.02% on OJBench, whereas TESSY achieves improvements of 11.25% and 6.68%.
Engineering Breakdown
Plain English
This paper addresses a critical failure mode in modern reasoning models: using synthetic data from stronger teacher models to fine-tune weaker student models often degrades performance instead of improving it. The authors identify that stylistic mismatch between teacher-generated data and the student's native style is the root cause. They propose TESSY (Teacher-Student Cooperation Data Synthesis), a framework where teacher and student models collaborate by alternately generating different parts of training sequences—the teacher handles reasoning-heavy tokens while the student contributes stylistic tokens. This hybrid generation approach lets the student learn advanced reasoning while staying true to its own expression patterns, avoiding the performance collapse seen in naive distillation.
Core Technical Contribution
The key innovation is recognizing that synthetic data quality for supervised fine-tuning isn't just about correctness—it's about stylistic consistency with the student model's intrinsic distribution. Previous approaches assumed any correct output from a stronger model would help a weaker model, but this breaks down for reasoning tasks where model personality and reasoning style matter. TESSY's core contribution is a collaborative token-level generation strategy where neither model generates complete sequences alone; instead, they partition the problem—teacher generates reasoning steps and logic, student generates connecting language and formatting. This interleaving mechanism is novel because it operates at sub-sequence granularity rather than document or task level, creating hybrid synthetic data that neither model could produce independently.
How It Works
The TESSY pipeline works as follows: given a reasoning task, the framework begins with a prompt or partial sequence. The teacher model generates a set of tokens for reasoning-critical positions (detection of which requires learned heuristics or explicit tagging—likely based on semantic importance), establishing the logical path forward. At stylistic positions (narrative transitions, rephrasing, format markers), the student model takes over and generates continuations that match its native style. The generation alternates back and forth through the sequence, with context windows expanding to include both teacher and student contributions. After generation completes, the full hybrid sequence becomes a training example for student fine-tuning. The framework requires careful token-level labeling or learned classifiers to partition reasoning versus style tokens, and likely includes a filtering step to reject incoherent interleaved sequences before they enter the SFT dataset.
Production Impact
For teams building reasoning models at scale, this directly addresses a major pain point: naive distillation pipelines often waste compute on generating and processing synthetic data that hurts model performance. Adopting TESSY would change the synthetic data generation phase of your training pipeline—instead of running the teacher model in isolation, you'd need bidirectional orchestration (teacher and student trading off generation). This requires more complex sampling logic and careful prompt engineering to define which tokens count as 'reasoning' versus 'style,' adding operational overhead. The upside is substantial: avoiding performance collapse means your compute investment in teacher model inference actually translates to student improvements, not wasted cycles. Trade-offs include 2-3x slower synthetic data generation (two models in the loop) versus single-pass teacher generation, latency during interleaved generation, and the need for labeled or classifierseparating semantic roles in your domain.
Limitations and When Not to Use This
The paper's applicability hinges on the assumption that reasoning and style can be cleanly separated at the token level, which may not hold for all task domains—for creative writing, poetry, or highly domain-specific reasoning where style and correctness are entangled, this partitioning breaks down. The approach also requires a sufficiently capable teacher model and a student model that has at least minimal reasoning ability; if the student is far below a performance threshold, its contributions may introduce noise that degrades the hybrid data quality. The paper doesn't address computational cost or provide wall-clock time comparisons, making it unclear if TESSY is practical for large-scale deployment with huge synthetic datasets. Additionally, the framework requires explicit design choices (which tokens are reasoning-critical) that may be task or model-specific, limiting generalization—there's no clear guidance on how to make these decisions for new domains without manual annotation or extensive tuning.
Research Context
This work builds on a decade of research in knowledge distillation and synthetic data generation, but identifies a failure mode specific to reasoning models (like large language models trained on code and math) that previous distillation work didn't address. It relates to concurrent work on model alignment and style-transfer in NLP, as well as emerging research on 'reasoning collapse' in distillation—where student models fail to acquire reasoning abilities from teachers despite learning surface patterns. The paper advances the intersection of distillation, curriculum learning, and multi-agent training, opening a research direction around adaptive task partitioning for synthetic data generation. Future work likely extends this to automatically discover reasoning-critical positions, apply TESSY across heterogeneous model families, and scale it to reasoning benchmarks like MATH-500 or AIME.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::