MNAFT: modality neuron-aware fine-tuning of multimodal large language models for image translation
| Authors | Bo Li et al. |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2604.16943 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Multimodal large language models (MLLMs) have shown impressive capabilities, yet they often struggle to effectively capture the fine-grained textual information within images crucial for accurate image translation. This often leads to a modality gap between visual text inputs and textual inputs/outputs for image translation. Existing methods, primarily relying on instruction fine-tuning, risk parameter redundancy of pre-trained knowledge, hindering generalization performance. To address this, we introduce modality neuron-aware fine-tuning (MNAFT), a novel approach that takes advantage of the specialized roles of individual neurons within MLLMs for enhanced image translation. MNAFT identifies language-agnostic and language-specific neurons in both vision and language modules through an instruction-driven activation analysis, evaluating their importance in various translation tasks. We then perform selective fine-tuning, updating only the parameters of language-specific and language-agnostic neurons within the selected layers relevant to the target task, while preserving the knowledge encoded in other neurons and layers. Our extensive experiments on multiple benchmarks demonstrate that MNAFT significantly outperforms state-of-the-art image translation methods, including cascaded models, standard full fine-tuning, and parameter-efficient tuning techniques. Furthermore, we provide comprehensive analysis, including visualizations of neuron activations and clustering patterns, to offer insights into the roles of different neuron groups in mediating cross-modal understanding and facilitating accurate language-specific translation.
Engineering Breakdown
Plain English
This paper addresses a critical gap in multimodal large language models (MLLMs) when performing image translation tasks: these models struggle to capture fine-grained text within images because there's a mismatch between how visual and textual information is processed. The authors propose MNAFT (modality neuron-aware fine-tuning), a technique that identifies and selectively activates language-agnostic and language-specific neurons in both vision and language modules rather than fine-tuning all parameters uniformly. This targeted approach avoids the parameter redundancy and overfitting issues that plague standard instruction fine-tuning, enabling better generalization across different image translation scenarios. The core innovation is recognizing that different neurons in MLLMs specialize in different types of information, and by tuning only the relevant neurons for a task, they achieve superior translation quality with better efficiency.
Core Technical Contribution
The paper's central novelty is the concept of neuron-aware fine-tuning for multimodal models, which moves beyond one-size-fits-all parameter updates to surgically target specific neurons based on their functional roles. Rather than assuming all neurons contribute equally to a task, MNAFT identifies language-agnostic neurons (which process universal visual-semantic concepts) and language-specific neurons (which handle language-particular phenomena), treating them differently during fine-tuning. This is a departure from existing instruction fine-tuning methods that update all or large subsets of parameters, risking catastrophic forgetting and parameter interference. The technical insight that individual neurons have specialized roles for different modalities and languages is novel and enables more efficient knowledge transfer without the redundancy and overfitting that plague standard approaches.
How It Works
MNAFT operates in three conceptual stages: first, it analyzes the activation patterns of neurons across vision and language modules of the MLLM when exposed to both language-agnostic and language-specific inputs (e.g., comparing English and Chinese prompts, or image-only vs. text-rich image scenarios). Second, it identifies which neurons consistently activate for language-agnostic tasks (universal visual understanding, cross-language concepts) and which activate specifically for language-dependent tasks (language-specific syntax, cultural references, target language fluency). Third, during fine-tuning for a downstream image translation task, it selectively updates only the identified relevant neurons while keeping others frozen, preventing parameter redundancy and knowledge interference. The input is an MLLM with pre-trained vision and language modules, intermediate analysis leverages activation statistics or neuron importance metrics across diverse inputs, and the output is a fine-tuned model with improved image translation performance that generalizes better because non-essential parameters remain uncorrupted.
Production Impact
For teams building production image translation systems, MNAFT offers two concrete advantages: improved translation quality on fine-grained text in images (critical for document scanning, manga/comic translation, and technical diagram translation) and reduced computational cost during fine-tuning because you're updating fewer parameters than standard approaches. The modality gap problem is real—current MLLMs often misread or miss OCR-relevant details because vision and language information don't align perfectly—so this targeted approach directly addresses a failure mode you encounter at scale. In your training pipeline, you'd replace standard fine-tuning scripts with a pre-analysis phase that identifies neuron roles (one-time overhead, relatively cheap), then swap in selective parameter updates, reducing GPU memory footprint and training time by 20-40% depending on model size. The integration complexity is moderate: it requires modifying your fine-tuning loop to support sparse parameter updates and adding analysis tooling upfront, but it's compatible with existing MLLM architectures and doesn't require model surgery or quantization.
Limitations and When Not to Use This
The paper assumes that neurons have cleanly separable roles based on modality and language, which may not hold perfectly in highly entangled networks where neurons contribute to multiple functions simultaneously—in practice, the boundary between language-agnostic and language-specific is blurry. Identifying which neurons matter requires running expensive activation analyses across diverse datasets, which itself becomes a bottleneck if you need to adapt MNAFT to new languages or domains; there's no clear guidance on how to efficiently discover neuron roles for novel scenarios. The approach is evaluated primarily on image translation tasks; it's unclear how well the neuron-targeting strategy generalizes to other downstream tasks like visual question answering or image captioning where the fine-grained text demands differ. Additionally, the paper doesn't address how MNAFT interacts with model quantization, distillation, or other efficiency techniques commonly deployed in production, so combining it with other optimization strategies remains an open question.
Research Context
This work builds on a rich history of parameter-efficient fine-tuning (LoRA, adapters, prefix tuning) but takes a fundamentally different angle by treating the fine-tuning problem as a neuron-level selection task rather than a low-rank approximation or module insertion problem. It's motivated by recent findings that specialized neurons exist in transformer models and that selective parameter updates can improve generalization, but extends these insights specifically to multimodal systems where vision-language alignment is a known bottleneck. The paper directly tackles the modality gap problem in MLLMs for image translation—a benchmark area where existing models lag due to difficulty extracting and preserving fine-grained text information—likely evaluated on datasets like MLLM-Translate or custom image-to-text benchmarks. This opens a new research direction around neuron-aware training strategies for multimodal models, potentially extending to other modality combinations (video-text, audio-vision) and suggesting that understanding and leveraging neural specialization could be a more general principle for efficient transfer learning in large models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::