Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset
| Authors | Dany Haddad et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2602.23335 |
| Download | |
| Categories | cs.HC, cs.AI, cs.IR |
Abstract
AI-powered scientific research tools are rapidly being integrated into research workflows, yet the field lacks a clear lens into how researchers use these systems in real-world settings. We present and analyze the Asta Interaction Dataset, a large-scale resource comprising over 200,000 user queries and interaction logs from two deployed tools (a literature discovery interface and a scientific question-answering interface) within an LLM-powered retrieval-augmented generation platform. Using this dataset, we characterize query patterns, engagement behaviors, and how usage evolves with experience. We find that users submit longer and more complex queries than in traditional search, and treat the system as a collaborative research partner, delegating tasks such as drafting content and identifying research gaps. Users treat generated responses as persistent artifacts, revisiting and navigating among outputs and cited evidence in non-linear ways. With experience, users issue more targeted queries and engage more deeply with supporting citations, although keyword-style queries persist even among experienced users. We release the anonymized dataset and analysis with a new query intent taxonomy to inform future designs of real-world AI research assistants and to support realistic evaluation.
Engineering Breakdown
Plain English
This paper presents Asta, a large-scale dataset of over 200,000 real user interactions with two AI-powered research tools (a literature discovery interface and a question-answering system) built on top of an LLM-based retrieval-augmented generation platform. The authors analyzed how researchers actually use these systems in production settings, finding that users submit longer and more complex queries than traditional search interfaces, and treat the AI system as a collaborative partner for tasks like content drafting. The dataset and interaction logs provide concrete evidence of how researchers engage with LLM-powered tools, enabling the field to move beyond speculation about real-world usage patterns. This work fills a significant gap in understanding human-AI interaction in scientific workflows, which has lacked empirical data from deployed systems at scale.
Core Technical Contribution
The core contribution is the Asta Interaction Dataset itself—a large-scale, longitudinal collection of 200,000+ real user queries and engagement logs from production-deployed scientific research tools, rather than lab experiments or synthetic interactions. The novelty lies in systematically characterizing how researchers actually adapt their behavior when using LLM-powered retrieval-augmented generation systems, including patterns of query complexity growth, task delegation, and engagement evolution over time. Unlike prior work that studied AI tool usage through surveys or small-scale studies, this dataset provides ground-truth behavioral telemetry from live deployments, enabling quantitative analysis of human-AI collaboration in scientific contexts. The authors' framework for analyzing query patterns and engagement behaviors provides a replicable methodology for understanding AI tool adoption across different user populations and research domains.
How It Works
The dataset was collected from two deployed tools within a single LLM-powered RAG platform: (1) a literature discovery interface that helps researchers find relevant papers, and (2) a question-answering interface for scientific queries. For each user interaction, the system logs the raw query text, interaction metadata (timestamps, user identifiers), system responses, and engagement signals (dwell time, reformulations, follow-up queries). The authors then apply analytical techniques to characterize query patterns—measuring length, syntactic complexity, domain-specificity—and track how these patterns evolve as individual users gain experience with the system over time. They also analyze behavioral signals like how often users refine queries, whether they delegate specific task types (synthesis, drafting, ideation) to the AI, and how engagement correlates with user tenure. The dataset captures the full interaction lifecycle, allowing reconstruction of user sessions and the reasoning behind query sequences.
Production Impact
For engineers building AI-powered research tools, this work provides empirical grounding for design decisions around query understanding, response generation, and iterative refinement interfaces. You can use patterns from this dataset to optimize your RAG system's relevance ranking and context retrieval specifically for scientific queries, since you now have evidence that researchers submit fundamentally different queries than web search users. The engagement metrics (query reformulation rates, session length distributions, task types delegated) can inform your system's user experience design—for example, whether to prioritize support for long multi-part queries, iterative refinement loops, or content generation capabilities. However, the production trade-off is that you need robust logging infrastructure to capture interaction telemetry at scale (200K+ queries across multiple tools), which requires careful data governance, privacy handling for researcher queries, and storage systems that can manage longitudinal user session data. The dataset also suggests you should instrument your RAG system to detect and support users' natural progression from simple queries to complex collaborative workflows.
Limitations and When Not to Use This
This dataset is specific to two particular tools within a single LLM-based RAG platform, so generalization to other research domains, tool architectures, or AI systems may be limited—the interaction patterns you see here might not hold for researchers using different types of AI systems (code generation, experimental design tools, etc.). The paper doesn't fully address privacy and ethical concerns around logging researcher queries, which may contain sensitive domain knowledge, unpublished ideas, or identifiable information; reproducing this work in regulated settings would face significant compliance challenges. The analysis is observational rather than experimental, so you cannot make causal claims about why users behave certain ways or what interventions would actually improve research outcomes—you see correlation between user tenure and query complexity, but not whether this is causation driven by learning or by selection bias (more sophisticated researchers staying longer). Additionally, the dataset likely exhibits survivorship bias and selection effects (which researchers adopt these tools, which stay engaged), limiting insights about why adoption fails or how to reach researchers who don't use AI assistance.
Research Context
This work builds on a growing body of research studying human-AI interaction and AI tool adoption in professional workflows, extending beyond lab-based studies to real-world deployment settings. It connects to broader research on retrieval-augmented generation systems and how LLMs behave when grounded in external knowledge bases, but shifts focus from system capabilities to actual user behavior and interaction patterns. The paper contributes to the emerging subfield of AI usability and human-computer interaction for research tools, where prior work has mostly relied on surveys or small user studies rather than large-scale interaction logs. This dataset opens pathways for future work on personalization (adapting the system to individual researcher expertise levels), query understanding at the intersection of natural language and domain-specific scientific language, and evaluating whether AI-assisted research tools actually improve research quality and productivity.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::