Visual-ERM: Reward Modeling for Visual Equivalence
| Authors | Ziyu Liu et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.13224 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
Vision-to-code tasks require models to reconstruct structured visual inputs, such as charts, tables, and SVGs, into executable or structured representations with high visual fidelity. While recent Large Vision Language Models (LVLMs) achieve strong results via supervised fine-tuning, reinforcement learning remains challenging due to misaligned reward signals. Existing rewards either rely on textual rules or coarse visual embedding similarity, both of which fail to capture fine-grained visual discrepancies and are vulnerable to reward hacking. We propose Visual Equivalence Reward Model (Visual-ERM), a multimodal generative reward model that provides fine-grained, interpretable, and task-agnostic feedback to evaluate vision-to-code quality directly in the rendered visual space. Integrated into RL, Visual-ERM improves Qwen3-VL-8B-Instruct by +8.4 on chart-to-code and yields consistent gains on table and SVG parsing (+2.7, +4.1 on average), and further strengthens test-time scaling via reflection and revision. We also introduce VisualCritic-RewardBench (VC-RewardBench), a benchmark for judging fine-grained image-to-image discrepancies on structured visual data, where Visual-ERM at 8B decisively outperforms Qwen3-VL-235B-Instruct and approaches leading closed-source models. Our results suggest that fine-grained visual reward supervision is both necessary and sufficient for vision-to-code RL, regardless of task specificity.
Engineering Breakdown
Plain English
Visual-ERM addresses a critical pain point in vision-to-code tasks: existing reward signals for reinforcement learning are too coarse or game-able, making it hard to train models that generate accurate visual outputs like rendered charts, tables, and SVGs. The paper proposes a multimodal generative reward model that evaluates code quality by rendering outputs and comparing them directly in visual space rather than relying on text rules or shallow embedding similarity. This approach provides fine-grained, interpretable feedback that captures visual discrepancies models would otherwise miss, solving the reward misalignment problem that has limited RL effectiveness in this domain.
Core Technical Contribution
The key innovation is framing reward modeling as a visual comparison problem rather than a text parsing or embedding similarity problem. Visual-ERM is a generative reward model that takes rendered outputs and provides feedback grounded in the visual rendering space, making it task-agnostic and resistant to reward hacking since it evaluates what actually appears on screen rather than proxy metrics. Unlike prior work using textual rules (which miss subtle visual issues) or coarse vision embeddings (which ignore fine-grained pixel-level differences), Visual-ERM directly models visual equivalence through a multimodal architecture that understands both visual and structural correctness simultaneously.
How It Works
The system operates in three stages: first, the vision-to-code model generates code (e.g., SVG, HTML, chart specification) from a visual input image; second, this code is rendered to produce an output image; third, the Visual-ERM reward model takes both the original image and rendered output as inputs and generates fine-grained feedback about visual discrepancies. The reward model is multimodal, meaning it processes visual information through a vision encoder while potentially incorporating structural/code information, allowing it to identify mismatches that pure visual metrics would miss. The output is not a single scalar reward but interpretable feedback that can identify specific visual errors (incorrect colors, wrong positioning, missing elements), enabling RL algorithms to learn from high-quality, gradient-friendly reward signals rather than sparse or misleading ones.
Production Impact
For teams building vision-to-code systems, this approach directly improves model quality in production by enabling effective RL fine-tuning after supervised training—a two-stage process that consistently outperforms SFT alone. The concrete benefit is higher visual fidelity in generated code outputs with fewer artifacts, which is critical for automating design-to-code pipelines, document digitization, and accessibility tools. Trade-offs include: computational overhead (rendering code + running inference on a reward model for every training step), data requirements (need paired examples of correct and incorrect visual outputs), and the complexity of integrating a rendering engine into your training loop. Latency impact is manageable since reward model inference happens offline during training, not at inference time, but total training time increases due to the rendering and evaluation overhead per sample.
Limitations and When Not to Use This
The paper assumes code can be reliably rendered to visuals, which breaks down for incomplete or syntactically invalid code—edge cases common in early RL training when models generate malformed outputs. Visual equivalence in pixel space may not align with semantic correctness; for example, two visually similar charts could have different underlying data or scale mismatches that aren't obvious from pixels alone. The approach requires a working rendering engine for the target format (SVG, HTML, etc.), limiting applicability to code types where deterministic rendering exists; dynamic or interactive code is harder to evaluate purely from static visual outputs. Reward hacking is mitigated but not eliminated—models could still learn to exploit quirks in how the reward model evaluates visuals rather than learning genuine visual correctness.
Research Context
This work builds directly on the broader theme of using RL to align LVLMs with complex, structured tasks where supervised learning alone plateaus, following the success of RLHF in language models. It advances the reward modeling literature by moving evaluation from text embeddings or rule-based metrics to the visual rendering domain, similar to how perceptual losses revolutionized image generation. The paper implicitly addresses the 'reward misalignment' problem that has hindered RL adoption in multimodal systems, opening a research direction around domain-specific reward models that evaluate output quality in the natural representation space (visual for vision tasks, acoustic for speech, etc.). This approach could inspire similar work in other structured generation domains where rendering or concrete output evaluation is possible.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::