Developing and evaluating a chatbot to support maternal health care
| Authors | Smriti Jha et al. |
| Year | 2026 |
| Field | AI / Agents |
| arXiv | 2603.13168 |
| Download | |
| Categories | cs.AI, cs.CL, cs.IR |
Abstract
The ability to provide trustworthy maternal health information using phone-based chatbots can have a significant impact, particularly in low-resource settings where users have low health literacy and limited access to care. However, deploying such systems is technically challenging: user queries are short, underspecified, and code-mixed across languages, answers require regional context-specific grounding, and partial or missing symptom context makes safe routing decisions difficult. We present a chatbot for maternal health in India developed through a partnership between academic researchers, a health tech company, a public health nonprofit, and a hospital. The system combines (1) stage-aware triage, routing high-risk queries to expert templates, (2) hybrid retrieval over curated maternal/newborn guidelines, and (3) evidence-conditioned generation from an LLM. Our core contribution is an evaluation workflow for high-stakes deployment under limited expert supervision. Targeting both component-level and end-to-end testing, we introduce: (i) a labeled triage benchmark (N=150) achieving 86.7% emergency recall, explicitly reporting the missed-emergency vs. over-escalation trade-off; (ii) a synthetic multi-evidence retrieval benchmark (N=100) with chunk-level evidence labels; (iii) LLM-as-judge comparison on real queries (N=781) using clinician-codesigned criteria; and (iv) expert validation. Our findings show that trustworthy medical assistants in multilingual, noisy settings require defense-in-depth design paired with multi-method evaluation, rather than any single model and evaluation method choice.
Engineering Breakdown
Plain English
This paper describes a production maternal health chatbot deployed in India that handles short, code-mixed queries from users with low health literacy in low-resource settings. The core challenge is that user queries are underspecified, span multiple languages, and require region-specific medical knowledge to route safely—particularly for high-risk pregnancy cases. The system combines stage-aware triage logic that identifies severity levels, hybrid retrieval over curated medical knowledge bases, and expert decision trees to route users appropriately. The approach was developed through a partnership between academic researchers, a health tech company, a nonprofit, and a hospital, addressing real deployment constraints around language diversity, medical safety, and limited connectivity.
Core Technical Contribution
The paper's novelty lies in a task-specific triage architecture designed for maternal health at deployment scale rather than a general-purpose LLM approach. The key insight is stage-aware routing: the system first classifies query severity and pregnancy stage before deciding between template-based responses, hybrid retrieval, or human expert escalation. This stagewise decomposition solves the partial symptom context problem by using what is available (age, pregnancy trimester, basic symptom keywords) to constrain the response space rather than requiring complete clinical information. The hybrid retrieval component combines dense retrieval over curated documents with sparse retrieval and regional context grounding—a pragmatic choice for low-connectivity environments where you need reliable, interpretable answers over hallucinated generative outputs.
How It Works
The system operates in a pipeline: (1) User input preprocessing handles code-mixed text (mix of Hindi/English/regional languages) and extracts stage signals (pregnant vs. postpartum, trimester if available). (2) A triage classifier assigns severity (routine vs. high-risk) and pregnancy stage based on keywords and intent recognition. (3) For high-risk cases, the system routes to expert-authored templates—hand-crafted response trees built by maternal health clinicians that handle dangerous symptoms like bleeding or preeclampsia signals. (4) For routine queries, hybrid retrieval activates: dense retrieval (embedding-based) searches over curated maternal health documents while sparse retrieval (BM25/keyword) acts as a safety filter. (5) Retrieved chunks are ranked by regional relevance (filtering for India-specific guidelines, drug availability, cultural context) and assembled into a response. (6) If confidence is low or the query involves symptoms outside the knowledge base, the system escalates to human triage. The output is a text message formatted for SMS delivery with explicit uncertainty markers when confidence is partial.
Production Impact
For engineers building similar systems, this demonstrates that domain-critical applications (especially healthcare) should avoid end-to-end LLM generation in favor of hybrid retrieval + structured routing. You'd implement a severity classifier as a lightweight model (could be a small BERT fine-tune or even rule-based if your domain is narrow) that gates access to different inference paths—this reduces hallucination risk and makes failures debuggable. The regional grounding component is practically critical: store your knowledge base with metadata tags (geography, language, guideline version, contraindication warnings) and filter retrieval results before ranking; this prevents giving a user Indian drug dosages if they're in a different country. For SMS-constrained deployment, plan for token budget per response (typically 150-200 SMS characters) and ensure your retrieval returns short, high-signal snippets rather than full documents. The partnership model matters: you'll need clinical validation from hospital staff, continuous monitoring via health outcomes (not just chat metrics), and a feedback loop to detect when the knowledge base becomes stale or region-specific guidance changes.
Limitations and When Not to Use This
The paper does not address the cold-start problem for new users: the system relies on users providing (or the system inferring) pregnancy stage and severity, but a first-time user may not know how to describe symptoms. The hybrid retrieval approach assumes you have a hand-curated knowledge base—this doesn't scale to rare pregnancy complications or emerging conditions (e.g., new infection patterns post-vaccination). The triage classifier is trained on labeled examples, but obtaining ground-truth labels requires clinical review, and the paper doesn't fully specify how label quality and inter-annotator agreement were ensured across different health providers. Deployment in truly low-resource settings (poor connectivity, voice-only interfaces, no SMS) isn't thoroughly evaluated—the system is described as phone-based but may have been primarily tested on data users. Finally, the paper (based on the provided abstract) doesn't measure clinical outcomes (did users follow advice safely?) or long-term system drift (does performance degrade as regional medical guidelines change?); evaluation appears to focus on retrieval quality and triage accuracy rather than user safety in practice.
Research Context
This work sits at the intersection of retrieval-augmented generation (RAG) for medical QA and human-centered AI deployment in low-resource settings. It builds on prior work in domain-specific chatbots and symptom checkers but differs by explicitly handling code-mixing and stage-aware context, which are under-explored in academic benchmarks. The approach is a reaction against generic LLM-based medical assistants (like Med-PaLM), which lack regional grounding and generate plausible-sounding but unsafe responses; instead, it recombines structured routing (from older clinical decision support systems) with modern retrieval methods. The partnership model with hospitals and nonprofits opens a research direction on how to close the gap between ML paper results and real clinical deployment—most health AI papers evaluate on held-out test sets, but this work implies evaluation should include feedback from actual users in the target geography over months of deployment.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::