Stable Behavior, Limited Variation: Persona Validity in LLM Agents for Urban Sentiment Perception
| Authors | Neemias B da Silva et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.28048 |
| Download | |
| Categories | cs.CL, cs.SI |
Abstract
Large Language Models (LLMs) are increasingly used as proxies for human perception in urban analysis, yet it remains unclear whether persona prompting produces meaningful and reproducible behavioral diversity. We investigate whether distinct personas influence urban sentiment judgments generated by multimodal LLMs. Using a factorial set of personas spanning gender, economic status, political orientation, and personality, we instantiate multiple agents per persona to evaluate urban scene images from the PerceptSent dataset and assess both within-persona consistency and cross-persona variation. Results show strong convergence among agents sharing a persona, indicating stable and reproducible behavior. However, cross-persona differentiation is limited: economic status and personality induce statistically detectable but practically modest variation, while gender shows no measurable effect and political orientation only negligible impact. Agents also exhibit an extremity bias, collapsing intermediate sentiment categories common in human annotations. As a result, performance remains strong on coarse-grained polarity tasks but degrades as sentiment resolution increases, suggesting that simple label-based persona prompting does not capture fine-grained perceptual judgments. To isolate the contribution of persona conditioning, we additionally evaluate the same model without personas. Surprisingly, the no-persona model sometimes matches or exceeds persona-conditioned agreement with human labels across all task variants, suggesting that simple label-based persona prompting may add limited annotation value in this setting.
Engineering Breakdown
Plain English
This paper investigates whether giving LLMs different personas (defined by gender, economic status, political orientation, and personality traits) actually makes them behave differently when analyzing urban scenes. The researchers created multiple agents for each persona and had them evaluate images from the PerceptSent dataset, measuring both consistency within personas and differences across personas. They found that agents with the same persona produced very similar responses, showing stable behavior, but surprisingly found that cross-persona differentiation was limited—economic status and personality traits didn't induce meaningful behavioral divergence as expected.
Core Technical Contribution
The core contribution is a rigorous empirical study quantifying the actual behavioral effects of persona prompting in multimodal LLMs, rather than assuming it works. Prior work assumed personas would create diverse outputs, but this paper systematically measures both within-persona consistency (how stable each persona is) and cross-persona variation (whether different personas actually produce different judgments). The technical novelty lies in the factorial experimental design spanning multiple demographic and personality dimensions, paired with analysis of reproducibility—showing that while persona prompting creates internal consistency, it fails to produce the expected diversity across different persona types. This reveals a gap between the theoretical promise of persona-based prompting and its actual behavioral effects.
How It Works
The experimental pipeline operates as follows: first, the researchers define a factorial set of personas combining four dimensions—gender (e.g., male, female), economic status (e.g., low, high income), political orientation (e.g., left, right), and personality traits. Second, they instantiate multiple LLM agents per persona by prepending persona descriptions to prompts. Third, each agent evaluates urban scene images from the PerceptSent dataset, generating sentiment judgments or descriptive outputs. Fourth, they measure two metrics: within-persona consistency (using metrics like agreement or similarity scores across agents with identical personas) and cross-persona variation (comparing outputs across different persona types). The multimodal LLM processes both image and persona context to generate judgments, and the analysis quantifies whether the model's behavior meaningfully diverges based on the persona prompt injection.
Production Impact
For production systems using LLMs as perception proxies in urban analysis, this finding is critical: persona prompting alone may not reliably create diverse behavioral outputs, limiting its use for simulating multiple human perspectives or creating fairness-aware decision systems. If you're building a system that assumes different personas will produce different urban sentiment annotations or recommendations, this research suggests you'll get stable-but-homogeneous outputs instead. The trade-off is clear—you gain reproducibility (consistent behavior per persona) at the cost of diversity (limited cross-persona differentiation). Practically, this means production teams should either invest in alternative approaches (e.g., fine-tuning separate models, data augmentation, or architectural changes) rather than relying on prompt injection alone, or they should validate empirically that persona prompting achieves the behavioral diversity their application requires. The compute cost of querying multiple personas is low, but the behavioral benefit appears limited.
Limitations and When Not to Use This
The paper has several important limitations: first, the abstract is truncated and doesn't specify the full extent of cross-persona variation or which persona dimensions did produce differentiation (e.g., did gender induce more variation than economic status?). Second, the study is limited to the PerceptSent dataset and urban scene analysis—results may not generalize to other domains, image types, or tasks where persona effects could be stronger. Third, the paper doesn't explore whether more sophisticated persona engineering (e.g., longer, richer persona descriptions; few-shot examples per persona; fine-tuning) could improve cross-persona differentiation, leaving open whether the limitation is fundamental or due to the prompting approach. Finally, the research doesn't address whether the stable within-persona behavior is actually desirable or whether it masks important natural human diversity that exists even among people with similar demographic profiles.
Research Context
This work addresses a growing concern in the LLM-as-human-proxy literature: the assumption that simple prompting techniques can reliably simulate human diversity. It builds on prior research in prompt engineering and persona-based generation, but critically examines these techniques empirically rather than taking their effectiveness for granted. The paper contributes to the evaluation and fairness literature by highlighting a gap between intended and actual behavior of LLMs under persona prompting. It opens a research direction into more robust methods for achieving behavioral diversity in multimodal models, potentially including techniques from domain adaptation, multi-task learning, or controlled generation to create more meaningful persona effects.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::