Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learning
| Authors | Jiajie Li et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.20116 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
Conventional fine-tuning on domain-specific datasets can inadvertently alter a model's pretrained multimodal priors, leading to reduced generalization. To address this, we propose Chain-of-Adaptation (CoA), an adaptation framework designed to integrate domain knowledge while maintaining the model's inherent reasoning and perceptual capabilities. CoA introduces a structured reasoning format that enhances domain alignment without sacrificing general multimodal competence by reinforcement learning. Experiments on standard surgical benchmarks, under both in-distribution and out-of-distribution settings, demonstrate that CoA achieves higher accuracy, stronger generalization, and more stable behavior than supervised fine-tuning. Furthermore, ablation studies confirm that CoA effectively preserves the model's core visual-language abilities, providing a reliable pathway for domain specialization in VLMs.
Engineering Breakdown
Plain English
This paper introduces Chain-of-Adaptation (CoA), a framework that adapts pretrained multimodal models to domain-specific tasks without eroding their general reasoning capabilities. The authors observed that standard fine-tuning on surgical datasets degrades a model's ability to generalize outside that domain, so they designed CoA to integrate domain knowledge through structured reasoning while preserving pretrained priors using reinforcement learning. Experiments on surgical benchmarks show CoA outperforms supervised fine-tuning in accuracy, generalization, and stability across both in-distribution and out-of-distribution test settings. Ablation studies confirm the framework successfully maintains the model's original multimodal competence while achieving stronger domain alignment.
Core Technical Contribution
The core innovation is a structured adaptation approach that decouples domain-specific learning from general capability preservation by introducing an explicit reasoning format as an intermediary between the pretrained model and task-specific outputs. Instead of directly modifying model weights through supervised fine-tuning (which typically catastrophically forgets general skills), CoA uses reinforcement learning to guide the model toward domain-aligned reasoning patterns while maintaining fidelity to its pretrained representations. This is fundamentally different from prior adaptation methods like LoRA or standard fine-tuning, which either freeze most weights or uniformly adjust them—CoA instead controls what the model reasons about without altering how it reasons. The structured reasoning format acts as a soft constraint, allowing the model to learn domain priorities without overwriting its multimodal priors.
How It Works
CoA operates by inserting a chain-of-thought-style reasoning layer between the frozen pretrained model encoder and the task-specific decoder. The input (e.g., a surgical image) passes through the pretrained multimodal encoder, which produces standard representations; these representations are then fed into a trainable reasoning module that generates explicit intermediate reasoning steps formatted for the surgical domain (e.g., 'identify anatomical region → assess tissue state → predict next action'). These reasoning outputs are then passed to a task head that produces the final prediction. During training, reinforcement learning signals optimize the reasoning module to produce domain-aligned reasoning patterns while the RL objective includes a regularization term that penalizes deviation from the pretrained model's original output distribution, ensuring the model doesn't forget general capabilities. At inference, this structured reasoning provides interpretability and maintains the generalization properties of the pretrained model while adapting to surgical specifics.
Production Impact
For production systems, CoA solves the critical problem of domain adaptation without capability loss, particularly important in high-stakes applications like medical imaging where a model must remain robust to distribution shifts while mastering specialized tasks. Rather than retraining from scratch or accepting degraded generalization, engineers could adopt CoA to fine-tune pretrained vision-language models on proprietary surgical datasets while retaining zero-shot capabilities on unseen surgical scenarios or even out-of-domain (non-surgical) multimodal tasks. The latency impact is moderate—the additional reasoning module adds a forward pass, likely 10-20% inference time overhead compared to direct fine-tuning, but this is traded for significantly better generalization. Integration complexity is moderate: you need an RL training loop (PPO or similar) rather than standard supervised learning, which requires reward model design and more careful hyperparameter tuning, but the pretrained backbone remains frozen, reducing memory requirements and enabling efficient multi-task fine-tuning on shared infrastructure.
Limitations and When Not to Use This
CoA assumes you have a pretrained multimodal model (e.g., CLIP-style vision-language models) and access to domain-specific labeled data to train the reasoning module—it won't help if you're starting from scratch or if the pretrained model is poorly aligned with your domain. The paper focuses exclusively on surgical imaging benchmarks; generalization to other high-stakes domains (pathology, radiology, autonomous driving) is untested and the structured reasoning format may need domain-specific redesign. The RL training loop introduces instability and hyperparameter sensitivity compared to supervised fine-tuning, and the paper does not thoroughly address failure modes when the RL objective conflicts with the domain task objective. Finally, the paper does not compare against other structured adaptation methods (e.g., prompt tuning, adapter modules with explicit constraints) or test on truly long-tail out-of-distribution scenarios, leaving unclear how much of the benefit comes from the structured reasoning versus the RL-based regularization.
Research Context
This work sits at the intersection of domain adaptation and instruction-following in multimodal models, building on insights from prompt engineering (where explicit reasoning improves task performance) and from catastrophic forgetting literature (where RL-based regularization preserves pretrained knowledge). It extends recent advances in vision-language models and structured reasoning techniques like chain-of-thought prompting, adapting them to the fine-tuning setting rather than just inference. The paper benchmarks on surgical imaging datasets (likely including datasets like CHOLEC80 or similar procedural video benchmarks) and demonstrates improvements over supervised baselines, contributing to the growing body of work on efficient domain adaptation for multimodal models. This opens a research direction toward combining structured reasoning with RL-based knowledge preservation, potentially applicable to other domains where catastrophic forgetting is expensive (medical, autonomous systems, robotics).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::