When RAG Chatbots Expose Their Backend: An Anonymized Case Study of Privacy and Security Risks in Patient-Facing Medical AI
| Authors | Alfredo Madrid-García & Miguel Rujas |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2605.00796 |
| Download | |
| Categories | cs.CR, cs.AI, cs.CL |
Abstract
Background: Patient-facing medical chatbots based on retrieval-augmented generation (RAG) are increasingly promoted to deliver accessible, grounded health information. AI-assisted development lowers the barrier to building them, but they still demand rigorous security, privacy, and governance controls. Objective: To report an anonymized, non-destructive security assessment of a publicly accessible patient-facing medical RAG chatbot and identify governance lessons for safe deployment of generative AI in health. Methods: We used a two-stage strategy. First, Claude Opus 4.6 supported exploratory prompt-based testing and structured vulnerability hypotheses. Second, candidate findings were manually verified using Chrome Developer Tools, inspecting browser-visible network traffic, payloads, API schemas, configuration objects, and stored interaction data. Results: The LLM-assisted phase identified a critical vulnerability: sensitive system and RAG configuration appeared exposed through client-server communication rather than restricted server-side. Manual verification confirmed that ordinary browser inspection allowed collection of the system prompt, model and embedding configuration, retrieval parameters, backend endpoints, API schema, document and chunk metadata, knowledge-base content, and the 1,000 most recent patient-chatbot conversations. The deployment also contradicted its privacy assurances: full conversation records, including health-related queries, were retrievable without authentication. Conclusions: Serious privacy and security failures in patient-facing RAG chatbots can be identified with standard browser tools, without specialist skills or authentication; independent review should be a prerequisite for deployment. Commercial LLMs accelerated this assessment, including under a false developer persona; assistance available to auditors is equally available to adversaries.
Engineering Breakdown
Plain English
This paper presents a security assessment methodology for patient-facing medical chatbots built on retrieval-augmented generation (RAG), which are increasingly deployed to provide accessible health information. The authors conducted a two-stage security audit using Claude Opus 4.6 for exploratory prompt-based testing followed by manual verification through browser network inspection and API payload analysis. The work identifies governance and deployment lessons specific to generative AI systems in healthcare, where security failures have direct patient safety implications. Rather than reporting specific vulnerabilities, the paper demonstrates a systematic approach to finding and validating security weaknesses in production RAG systems without destructive testing.
Core Technical Contribution
The core contribution is a practical, non-destructive security assessment framework specifically designed for medical RAG chatbots that combines AI-assisted vulnerability discovery with manual verification. Rather than proposing a novel algorithm or architecture, the authors introduce a reproducible two-stage methodology: using Claude Opus 4.6 for hypothesis generation around security flaws, then systematically validating findings through developer tools and traffic inspection. This addresses a critical gap in the literature—while RAG systems are widely deployed in healthcare, there is limited published guidance on how to audit them for security vulnerabilities without causing harm. The framework treats the chatbot as a black box accessible through normal user interactions, making it practical for real-world assessment of deployed systems.
How It Works
The assessment uses a two-stage pipeline architecture. In Stage 1, Claude Opus 4.6 acts as an exploratory agent, generating prompt-based test cases designed to probe for common security weaknesses in RAG systems—such as prompt injection, information leakage, or retrieval bypass—by analyzing the chatbot's expected architecture and known vulnerability patterns. The LLM generates structured hypotheses about potential vulnerabilities rather than executing attacks directly. In Stage 2, researchers manually verify each candidate finding by examining browser-visible artifacts: network traffic captured through Chrome Developer Tools, API request/response payloads, and the chatbot's actual behavior under the hypothesized attack conditions. This staged approach separates hypothesis generation (which is fast and systematic) from verification (which is careful and manual), reducing false positives and ensuring that only confirmed vulnerabilities are reported. The methodology is non-destructive because it observes system behavior without modifying backend state, logs, or data—it only reads what a normal user would see.
Production Impact
For teams building medical chatbots or any sensitive domain RAG systems, this assessment framework provides a concrete playbook for security validation before production deployment or as an ongoing audit tool. Instead of relying on external penetration testers or hoping internal security reviews catch issues, engineers can use Claude or similar models to systematically generate plausible attack scenarios, then assign junior engineers to verify them—reducing both cost and time-to-security-validation. The approach is practical for production systems because it works through the same interface users interact with (the chatbot UI) and requires only standard browser tools, meaning it can run against live systems without coordination with backend teams or risk of accidental downtime. The trade-off is that this methodology is bounded by what an external user can observe and infer; it will not catch vulnerabilities in backend database queries, access control logic, or infrastructure that are invisible to the client. Organizations should adopt this as part of a layered security strategy, not as a complete replacement for code review, threat modeling, or infrastructure audits.
Limitations and When Not to Use This
The assessment is limited to vulnerabilities observable through the user-facing chatbot interface and network traffic; it cannot detect backend security flaws in database queries, API authentication, or infrastructure architecture that are not reflected in the chatbot's behavior. The methodology relies on Claude Opus 4.6 to generate security hypotheses, which means it is only as comprehensive as the model's training data on healthcare chatbot vulnerabilities—emerging or novel attack vectors outside the model's knowledge will be missed. The paper does not provide quantitative benchmarks (e.g., vulnerability detection rate, false positive rate compared to manual testing or other automated tools), making it difficult to measure the effectiveness of the approach against other assessment methods. Additionally, the work is presented as an anonymized case study of one chatbot; generalization to other RAG architectures, deployment patterns, or domain-specific applications remains open, and the specific findings may not transfer if the target system uses different LLMs, retrieval backends, or prompt engineering practices.
Research Context
This work sits at the intersection of two active research areas: (1) RAG system robustness and (2) AI safety and governance in high-stakes domains. It builds on prior work on prompt injection attacks and LLM robustness, but adapts those insights to the specific constraints and trust requirements of healthcare. The paper contributes to the emerging field of AI governance by providing practical, reproducible assessment techniques that can help organizations meet regulatory expectations (e.g., FDA guidance on AI/ML in medical devices, which increasingly requires pre-deployment safety and security validation). This methodology opens a research direction toward systematic, AI-assisted security assessment frameworks for deployed LLM systems, where the audit tool itself is an LLM—creating a new class of meta-security problems (how do you verify that your security auditor LLM is thorough and unbiased?).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::