SWE-chat: Coding Agent Interactions From Real Users in the Wild
| Authors | Joachim Baumann et al. |
| Year | 2026 |
| HF Upvotes | 10 |
| arXiv | 2604.20779 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
AI coding agents are being adopted at scale, yet we lack empirical evidence on how people actually use them and how much of their output is useful in practice. We present SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild. The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls. SWE-chat is a living dataset; our collection pipeline automatically and continually discovers and processes sessions from public repositories. Leveraging SWE-chat, we provide an initial empirical characterization of real-world coding agent usage and failure modes. We find that coding patterns are bimodal: in 41% of sessions, agents author virtually all committed code ("vibe coding"), while in 23%, humans write all code themselves. Despite rapidly improving capabilities, coding agents remain inefficient in natural settings. Just 44% of all agent-produced code survives into user commits, and agent-written code introduces more security vulnerabilities than code authored by humans. Furthermore, users push back against agent outputs -- through corrections, failure reports, and interruptions -- in 44% of all turns. By capturing complete interaction traces with human vs. agent code authorship attribution, SWE-chat provides an empirical foundation for moving beyond curated benchmarks towards an evidence-based understanding of how AI agents perform in real developer workflows.
Engineering Breakdown
Plain English
SWE-chat is the first large-scale dataset of real coding agent interactions collected from open-source developers, containing 6,000 sessions with over 63,000 user prompts and 355,000 agent tool calls. The authors built an automated pipeline that continuously discovers and processes coding sessions from public repositories, creating a living dataset that grows over time. Their empirical analysis reveals that coding agent usage is bimodal: in 41% of sessions agents author virtually all committed code ("vibe coding"), suggesting two distinct user behaviors and adoption patterns. This dataset and characterization provide concrete evidence about how developers actually use AI coding assistants and where agents fail in practice.
Core Technical Contribution
The core contribution is SWE-chat itself—the first large-scale, continuously-updated dataset of real-world coding agent interactions collected from developers in production environments rather than controlled experiments or crowdsourced benchmarks. The novelty is both in the dataset construction (an automated pipeline that discovers sessions from public repositories at scale) and in the empirical characterization it enables: identifying bimodal usage patterns and concrete failure modes that wouldn't be visible in synthetic benchmarks. Prior work relied on either small curated datasets, synthetic tasks, or lab studies; SWE-chat provides naturalistic behavioral data at scale. The living dataset approach, where the collection pipeline continuously processes new sessions, enables ongoing empirical research as coding agent technology evolves.
How It Works
The system operates in two phases: collection and analysis. The collection pipeline automatically monitors public repositories, identifies sessions where users interacted with coding agents, and extracts structured data including prompts, agent responses, tool calls, and code commits. For each session, they capture the full interaction history—what the user asked, which tools the agent invoked (e.g., code search, file editing, test execution), the agent's responses, and whether the code was ultimately committed to the repository. The analysis phase processes these 6,000 sessions to characterize usage patterns, identifying metrics like the ratio of agent-authored vs. user-authored code per session, frequency of different tool use patterns, and correlation between agent outputs and committed code. The dataset is structured to enable longitudinal analysis, allowing researchers to track how agent behavior and user interactions change over time as models improve.
Production Impact
For teams building coding agents, this dataset and empirical characterization directly inform model evaluation and real-world expectations. Rather than relying on synthetic benchmarks (which often overestimate agent capability), engineers can now validate their agents against actual usage patterns and understand that bimodal behavior means their model may serve fundamentally different user segments—some fully offloading coding to the agent, others using it for targeted assistance. The concrete failure modes identified enable targeted improvements: if the data shows agents frequently fail at specific tool combinations or code domain patterns, teams can prioritize retraining on those scenarios. The living dataset approach provides a continuous feedback loop: once you integrate your agent with real users, you can contribute anonymized sessions back to SWE-chat or build internal versions, then analyze what's actually breaking to drive the next iteration. The main trade-off is privacy: collecting real developer data at scale requires robust anonymization and contributor consent, adding operational complexity compared to synthetic benchmarks.
Limitations and When Not to Use This
The dataset is biased toward open-source developers and public repositories, which represent a narrow slice of real-world coding work—proprietary codebases, domain-specific languages, and enterprise workflows may show very different agent usage patterns and failure modes. The "living dataset" approach introduces temporal non-stationarity: agent behavior and user expectations shift as models improve, making longitudinal comparisons difficult and requiring careful versioning of the dataset. The paper doesn't deeply explore causality—it shows correlation between vibe coding (agent authoring most code) and other patterns, but doesn't explain why some users prefer this mode or how it affects code quality and maintainability long-term. Finally, the dataset captures tool calls and commits but may miss important context about user satisfaction, frustration, or the invisible costs of reviewing and debugging agent output—metrics that aren't directly measurable from repository data.
Research Context
This work addresses a gap in the AI systems research literature: while large language models and code generation have advanced rapidly, there's been minimal empirical data on how real developers actually use coding agents in production. It builds on prior work in human-computer interaction for developer tools and dataset construction for code (like CodeSearchNet and GitHub-scale studies), but shifts the lens from "how good is the agent" to "how do people really use it." The dataset complements recent work on coding agent benchmarks (like SWE-bench) which focus on synthetic task performance; SWE-chat provides naturalistic behavioral ground truth. This research opens directions on understanding human-agent coding collaboration, measuring practical impact on developer productivity, and studying how bimodal usage affects code quality and team dynamics.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::