Can LLMs Understand the Impact of Trauma? Costs and Benefits of LLMs Coding the Interviews of Firearm Violence Survivors
| Authors | Jessica H. Zhu et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.16132 |
| Download | |
| Categories | cs.CL, cs.AI |
Abstract
Firearm violence is a pressing public health issue, yet research into survivors' lived experiences remains underfunded and difficult to scale. Qualitative research, including in-depth interviews, is a valuable tool for understanding the personal and societal consequences of community firearm violence and designing effective interventions. However, manually analyzing these narratives through thematic analysis and inductive coding is time-consuming and labor-intensive. Recent advancements in large language models (LLMs) have opened the door to automating this process, though concerns remain about whether these models can accurately and ethically capture the experiences of vulnerable populations. In this study, we assess the use of open-source LLMs to inductively code interviews with 21 Black men who have survived community firearm violence. Our results demonstrate that while some configurations of LLMs can identify important codes, overall relevance remains low and is highly sensitive to data processing. Furthermore, LLM guardrails lead to substantial narrative erasure. These findings highlight both the potential and limitations of LLM-assisted qualitative coding and underscore the ethical challenges of applying AI in research involving marginalized communities.
Engineering Breakdown
Plain English
This paper evaluates whether open-source large language models can automatically code qualitative interview data from firearm violence survivors—a task that normally requires manual human analysis by researchers. The authors worked with 21 Black interview participants and tested LLMs on inductive coding, a qualitative research method where themes emerge from the data rather than being predefined. The core challenge is determining if these models can accurately and ethically capture sensitive experiences from vulnerable populations without losing nuance or introducing bias. This matters because manual coding is extremely time-consuming, so automating it could enable researchers to scale qualitative studies on public health crises.
Core Technical Contribution
The paper's primary contribution is a systematic evaluation framework for assessing whether open-source LLMs can perform inductive coding—a social science research task that has never been systematically automated before. Rather than building a new model architecture, the authors establish methodology for validating that LLM-generated codes match human researcher coding, with explicit attention to whether vulnerable populations' experiences are accurately represented. The technical novelty lies in designing a rigorous validation approach that combines code accuracy metrics with qualitative assessment of whether themes are faithfully captured, bridging the gap between NLP evaluation and social science standards. This work essentially translates a labor-intensive manual research process into an LLM task while maintaining scientific rigor.
How It Works
The system takes transcribed interview text from firearm violence survivors as input and passes it to an open-source LLM with a prompt structured for inductive coding—asking the model to identify emerging themes without predefined codes. The LLM processes the narrative text and outputs candidate codes (short category labels) along with evidence snippets from the interviews that support each code. These LLM-generated codes are then compared against codes manually produced by human researchers using standard qualitative research validation methods (likely inter-rater reliability measures and qualitative assessment of code validity). The pipeline iterates through refinement cycles where discrepancies between human and model codes are analyzed to understand whether the LLM captured genuine themes or missed context-specific nuances that matter for vulnerable populations. The final output is a validated set of codes that researchers can use to organize and analyze the full corpus of interviews.
Production Impact
For research organizations and public health teams, this approach could reduce the 40-80 hours typically needed to manually code 20-30 interviews down to a few hours of LLM processing plus targeted human validation of high-uncertainty codes. A production system would integrate open-source LLM inference (likely running locally or on-premise for privacy with sensitive health data) into a qualitative research pipeline, with a human-in-the-loop validation interface where researchers review and approve/reject model-generated codes. The compute cost is minimal compared to manual labor—a single GPU inference pass costs under $1 per interview at scale—making this economically viable for under-resourced research teams studying public health crises. Key trade-offs include: you still need domain experts to validate outputs (it's not fully automated), the model may miss cultural or contextual nuances specific to Black communities' experiences, and you need careful prompt engineering to avoid imposing researcher bias through the coding prompt.
Limitations and When Not to Use This
The paper's scope appears limited to 21 participants, which is a small validation set—real production deployment would need validation on much larger datasets across different researcher teams and communities to confirm the approach generalizes. A critical gap is whether open-source LLMs adequately preserve the cultural and contextual specificity of Black participants' experiences with firearm violence, or whether they flatten narratives into generic codes that miss community-specific meaning. The authors don't appear to address how the model handles code drift (when new themes emerge that weren't in the initial code set) or how to handle disagree between human coders and the model in cases where humans themselves don't fully agree. There's also an unstated risk: if researchers rely too heavily on automated coding, they might reduce the deep qualitative reflection that often generates new insights—automation could shift the research practice in ways that harm discovery.
Research Context
This work sits at the intersection of computational social science and qualitative research automation, building on recent success in using LLMs for document classification and NLP tasks while extending them to structured social science methodology. It responds to a gap in applied NLP: most LLM research focuses on benchmark datasets (SQuAD, GLUE, etc.), but very little work applies LLMs to labor-intensive real-world research tasks that affect public health or social science. The paper implicitly critiques the assumption that LLMs are only useful for standard NLP tasks, and opens a research direction in automating qualitative research at scale—potentially enabling researchers with limited resources to tackle larger studies of vulnerable communities. This could inspire similar work in automating other qualitative methods (grounded theory, phenomenology) or scaling qualitative research in fields like education, criminal justice, and community health.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::