Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language
| Authors | Yi Zhong et al. |
| Year | 2026 |
| HF Upvotes | 20 |
| arXiv | 2604.19667 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
At present, executable visual workflows have emerged as a mainstream paradigm in real-world industrial deployments, offering strong reliability and controllability. However, in current practice, such workflows are almost entirely constructed through manual engineering: developers must carefully design workflows, write prompts for each step, and repeatedly revise the logic as requirements evolve-making development costly, time-consuming, and error-prone. To study whether large language models can automate this multi-round interaction process, we introduce Chat2Workflow, a benchmark for generating executable visual workflows directly from natural language, and propose a robust agentic framework to mitigate recurrent execution errors. Chat2Workflow is built from a large collection of real-world business workflows, with each instance designed so that the generated workflow can be transformed and directly deployed to practical workflow platforms such as Dify and Coze. Experimental results show that while state-of-the-art language models can often capture high-level intent, they struggle to generate correct, stable, and executable workflows, especially under complex or changing requirements. Although our agentic framework yields up to 5.34% resolve rate gains, the remaining real-world gap positions Chat2Workflow as a foundation for advancing industrial-grade automation. Code is available at https://github.com/zjunlp/Chat2Workflow.
Engineering Breakdown
Plain English
This paper addresses the problem that visual workflows (executable automation sequences used widely in industry) are currently built entirely by hand—developers write prompts for each step, design logic, and iterate repeatedly when requirements change. The authors introduce Chat2Workflow, a benchmark dataset and agentic framework that uses large language models to automatically generate these executable workflows directly from natural language descriptions. The key innovation is a robust agent that handles the multi-round interaction process and mitigates execution errors that occur during workflow generation. This automates what has historically been a labor-intensive, error-prone manual engineering task.
Core Technical Contribution
The core novelty is combining two things: (1) Chat2Workflow, the first large-scale benchmark dataset of real-world visual workflows paired with natural language specifications, enabling measurement of LLM capability at workflow generation, and (2) an agentic framework designed specifically to handle the iterative, error-prone nature of converting language to executable workflows. Unlike prior work that focuses on code generation or simple task automation, this approach targets the full lifecycle of workflow construction—including error recovery, logic refinement, and step-by-step prompt engineering. The agent architecture is built to diagnose and correct execution failures at each stage rather than generating perfect workflows on the first pass.
How It Works
The system takes a natural language description of a desired workflow as input, then an LLM-powered agent iteratively generates the corresponding visual workflow definition (e.g., a DAG or step-by-step execution plan). At each step, the agent proposes workflow components, attempts to execute them, and receives feedback on whether execution succeeded or failed. When errors occur—such as incompatible data types between steps, missing parameters, or logical inconsistencies—the agent diagnoses the failure and regenerates the problematic portion. This multi-round loop continues until the workflow executes successfully end-to-end. The benchmark dataset (Chat2Workflow) provides ground-truth workflows and natural language descriptions, allowing evaluation of how accurately agents can reconstruct the intended workflow structure and logic. The framework likely uses techniques like few-shot prompting, chain-of-thought reasoning, and execution feedback to improve generation quality across iterations.
Production Impact
In production, this approach could dramatically reduce the engineering cost of workflow creation. Currently, building a non-trivial automation workflow (e.g., data pipeline, business process orchestration, or multi-step API integration) requires domain expertise, manual prompt engineering, and weeks of iteration. With Chat2Workflow, engineers could describe what they want in plain language and have an agent generate 80-90% of the boilerplate, then manually review and refine edge cases. This would accelerate time-to-production and lower barriers to entry for non-technical users. The trade-off is that the system introduces additional LLM inference latency (each validation loop calls the model), higher token costs during development (multi-round generation), and a dependency on the agent's ability to recover from errors—so workflows with novel or ambiguous requirements may still fail and require manual debugging. Integration requires monitoring execution feedback and providing clear error messages back to the agent, adding observability complexity.
Limitations and When Not to Use This
The paper assumes that workflows can be completely specified and debugged through natural language alone; in practice, many production workflows require domain-specific knowledge, data schema awareness, or business rules that may be implicit or context-dependent. The approach is also limited by the underlying LLM's reasoning ability—if the model cannot understand the natural language description or diagnose execution errors reliably, the multi-round loop may get stuck or diverge. Additionally, the benchmark is built from existing workflows, so the approach may not generalize to novel workflow patterns, custom integrations, or adversarial edge cases not represented in the training data. The paper does not appear to address how well the framework scales to very large, complex workflows with dozens of steps and conditional branching, or how it handles workflows that require human-in-the-loop decisions mid-execution.
Research Context
This work builds on a growing body of research in LLM-assisted automation and code generation (like Codex, GitHub Copilot, and Chain-of-Thought prompting), but applies those ideas to the specific domain of visual workflow construction rather than source code. It aligns with the trend of using LLMs as agents that can iteratively refine outputs through execution feedback, similar to work on LLM-based planning and reinforcement learning from human feedback. The Chat2Workflow benchmark contributes to the broader ecosystem of ML datasets by providing the first large-scale, real-world evaluation set for workflow generation—analogous to how datasets like HumanEval advanced code generation evaluation. The agentic error-recovery framework opens a research direction into making LLM-based automation more robust and production-ready by embracing iterative refinement rather than expecting single-pass perfection.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::