Scientific Graphics Program Synthesis via Dual Self-Consistency Reinforcement Learning
| Authors | Juekai Lin et al. |
| Year | 2026 |
| HF Upvotes | 5 |
| arXiv | 2604.06079 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Graphics Program Synthesis is pivotal for interpreting and editing visual data, effectively facilitating the reverse-engineering of static visuals into editable TikZ code. While TikZ is the de facto standard for scientific schematics due to its programmatic flexibility, its requirement for rigorous spatial precision presents a significant challenge for Multimodal Large Language Models. Progress is currently stifled by two primary gaps: (1) Data Quality Gap: existing image-TikZ corpora often lack strict executability and reliable visual alignment; (2) Evaluation Gap: a lack of benchmarks for both structural and visual fidelity. To address these, we present a closed-loop framework featuring: SciTikZ-230K, a large-scale, high-quality dataset from our Execution-Centric Data Engine covering 11 diverse scientific disciplines; SciTikZ-Bench, a multifaceted benchmark spanning from basic geometric constructs to intricate hierarchical schematics to evaluate both visual fidelity and structural logic. To further broaden the scope of visual-code optimization methodology, we introduce a novel Dual Self-Consistency Reinforcement Learning optimization paradigm, which utilizes Round-Trip Verification to penalize degenerate code and boost overall self-consistency. Empowered by these, our trained model SciTikZer-8B achieves state-of-the-art performance, consistently outperforming proprietary giants like Gemini-2.5-Pro and massive models like Qwen3-VL-235B-A22B-Instruct.
Engineering Breakdown
Plain English
This paper tackles the problem of converting static images of scientific diagrams into executable TikZ code, which is a standard format for creating vector graphics in academic papers. The authors identified two major bottlenecks: existing image-to-TikZ datasets have poor quality and don't reliably execute, and there's no proper benchmark to evaluate both the structural correctness and visual accuracy of generated code. They built SciTikZ-230K, a large-scale curated dataset of 230,000 examples using an Execution-Centric Data Engine that ensures every generated TikZ snippet actually compiles and visually matches the input image. This closed-loop framework directly addresses why multimodal language models struggle with the spatial precision required for graphics programming.
Core Technical Contribution
The core innovation is the Execution-Centric Data Engine—a system that enforces strict executability and visual fidelity at data collection time, not post-hoc evaluation. Rather than scraping or synthetically generating image-TikZ pairs and hoping they align, the engine closes the loop: it generates TikZ code, compiles it, renders the output, and only accepts data points where the rendered output matches the original image within defined tolerances. This is fundamentally different from prior work that treated data quality as a downstream problem in evaluation. The paper also contributes SciTikZ-230K, a benchmark-quality dataset covering 11 different diagram types, and establishes evaluation metrics that measure both structural correctness (does the code compile?) and visual fidelity (does the output look right?).
How It Works
The pipeline starts with collecting or generating image-TikZ pairs from scientific papers and diagrams. Instead of accepting these raw, the Execution-Centric Data Engine takes each pair and runs TikZ compilation on the code snippet in a sandboxed environment—if compilation fails, the example is rejected. Next, the engine renders the compiled TikZ output as a raster image and compares it pixel-by-pixel or using perceptual similarity metrics against the original input image. Only pairs that pass both the execution check and the visual alignment check are added to the dataset. This creates a virtuous cycle: when training multimodal models on SciTikZ-230K, they learn from examples where the code is guaranteed to work and the visual output matches reality. The framework then enables researchers to evaluate new models not just on whether generated code compiles, but on whether it produces visually faithful output.
Production Impact
For teams building document processing or scientific paper authoring systems, this work removes a major pain point: the inability to reliably convert raster diagrams back into editable vector code. In a production pipeline, engineers could use a finetuned multimodal model on SciTikZ-230K to automatically reconstruct editable TikZ from PDF scans or image uploads, dramatically reducing manual re-creation time for scientific documents. The Execution-Centric Data Engine approach is also reusable: teams building code synthesis systems for any domain (SQL, Python, LaTeX) could adopt the same closed-loop filtering strategy to improve dataset quality. The trade-off is computational overhead during data curation—you must compile and render every candidate example, adding significant preprocessing time and complexity. However, this front-loaded cost directly improves model robustness and reduces post-deployment failures where generated code doesn't compile or produces wrong output.
Limitations and When Not to Use This
The paper assumes all diagrams can be represented in TikZ's feature set, but real scientific papers contain complex diagrams (photorealistic images, 3D plots, embedded photographs) that TikZ cannot reproduce. The evaluation relies on visual similarity metrics, which may not capture semantic correctness—code could render to pixel-perfect output but encode the diagram's structure wrong, making it non-editable. The dataset covers only 11 diagram types, leaving open questions about generalization to rare or novel diagram styles not present in training data. Additionally, the paper does not address the inference latency or cost of multimodal models at scale, so it's unclear whether using large language models for this task is practical compared to specialized graphics recognition pipelines that might be faster and cheaper.
Research Context
This work builds on the recent success of multimodal large language models (like GPT-4V) in understanding images, but identifies that vision-language pretraining alone is insufficient for the precision required in graphics programming. It extends prior code synthesis research (which typically focused on programming languages like Python or SQL) into the visual domain, where spatial accuracy is as important as syntactic correctness. The paper contributes to a growing trend of curating high-quality, execution-validated datasets for code generation—similar in spirit to works that filter training data for correctness. It opens a new research direction: closed-loop data engines for any task where ground truth can be deterministically verified, potentially influencing how future ML datasets are built across domains like hardware design, CAD, and scientific computing.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::