No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpus
| Authors | Hitesh Mehta et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.16275 |
| Download | |
| Categories | cs.CL |
Abstract
This paper explores the response of Large Language Models (LLMs) to user prompts with different degrees of politeness and impoliteness. The Politeness Theory by Brown and Levinson and the Impoliteness Framework by Culpeper form the basis of experiments conducted across three languages (English, Hindi, Spanish), five models (Gemini-Pro, GPT-4o Mini, Claude 3.7 Sonnet, DeepSeek-Chat, and Llama 3), and three interaction histories between users (raw, polite, and impolite). Our sample consists of 22,500 pairs of prompts and responses of various types, evaluated across five levels of politeness using an eight-factor assessment framework: coherence, clarity, depth, responsiveness, context retention, toxicity, conciseness, and readability. The findings show that model performance is highly influenced by tone, dialogue history, and language. While polite prompts enhance the average response quality by up to ~11% and impolite tones worsen it, these effects are neither consistent nor universal across languages and models. English is best served by courteous or direct tones, Hindi by deferential and indirect tones, and Spanish by assertive tones. Among the models, Llama is the most tone-sensitive (11.5% range), whereas GPT is more robust to adversarial tone. These results indicate that politeness is a quantifiable computational variable that affects LLM behaviour, though its impact is language- and model-dependent rather than universal. To support reproducibility and future work, we additionally release PLUM (Politeness Levels in Utterances, Multilingual), a publicly available corpus of 1,500 human-validated prompts across three languages and five politeness categories, and provide a formal supplementary analysis of six falsifiable hypotheses derived from politeness theory, empirically assessed against the dataset.
Engineering Breakdown
Plain English
This paper investigates how five major LLMs (Gemini-Pro, GPT-4o Mini, Claude 3.7 Sonnet, DeepSeek-Chat, Llama 3) respond differently when users employ polite versus impolite language across three languages (English, Hindi, Spanish). The researchers evaluated 22,500 prompt-response pairs using an eight-factor assessment framework measuring coherence, clarity, depth, responsiveness, context retention, toxicity, conciseness, and readability. The core finding—though the abstract cuts off—is that model performance is highly influenced by politeness levels in user prompts, suggesting that social conventions encoded in language affect LLM behavior in measurable ways. This work bridges linguistic politeness theory (Brown and Levinson, Culpeper) with empirical LLM evaluation across multiple models and languages.
Core Technical Contribution
The technical novelty here is applying formal linguistic politeness and impoliteness frameworks to systematically measure LLM response quality variation across politeness conditions. Rather than studying LLMs in isolation, the authors operationalized Brown and Levinson's Politeness Theory and Culpeper's Impoliteness Framework as measurable input variables, then correlated them against eight distinct output quality dimensions. This is the first large-scale cross-model, cross-language study quantifying how non-semantic linguistic properties (politeness markers) affect LLM output quality. The eight-factor assessment framework itself is a contribution—it moves beyond single-metric evaluation to capture nuanced aspects of response quality that matter in production systems.
How It Works
The experiment structure works as follows: (1) Generate or select 22,500 prompt-response pairs across three interaction history contexts (raw prompts, polite variants, impolite variants) for each of the five LLMs. (2) For each prompt, the authors vary the politeness level while keeping the semantic content approximately constant—this isolates the effect of politeness markers alone. (3) Feed these prompts to the five models across three languages, capturing raw responses. (4) Evaluate each response on eight dimensions: coherence (logical consistency), clarity (understandability), depth (informativeness), responsiveness (question relevance), context retention (maintaining conversation history), toxicity (harmful content), conciseness (avoiding verbosity), and readability (structural quality). (5) Aggregate scores by model, language, and politeness level to identify patterns. The key technical insight is treating politeness as a controlled variable in prompt engineering rather than as a confound, allowing causal inference about how social language features influence model behavior.
Production Impact
undefined
Limitations and When Not to Use This
The paper does not explain the causal mechanism—we don't know whether politeness improves clarity of thought, whether models are pattern-matching to training data biases, or whether higher politeness simply correlates with longer, more detailed prompts. It's also unclear whether the eight-factor framework has been validated for cross-cultural applicability; politeness, toxicity, and clarity carry different meanings across Hindi, Spanish, and English speech communities, potentially conflating cultural preferences with model robustness. The study is limited to five models and three languages—results may not generalize to smaller models, multilingual models, or languages with different grammatical politeness systems (e.g., Japanese keigo). Finally, the paper likely assumes static prompts; real-world applications involve multi-turn conversations where politeness history and context drift may compound in unpredictable ways. No detail is given on whether prompts were human-written or synthetic, which affects ecological validity.
Research Context
This work builds directly on Brown and Levinson's foundational 1987 Politeness Theory and Culpeper's 2005 Impoliteness Framework—linguistic models that predate LLMs. It sits at the intersection of NLP pragmatics research (studying how context shapes meaning) and LLM evaluation benchmarks (like HELM, which assess multi-dimensional quality). Prior work examined LLM toxicity (Welbl et al., 2021) and fairness across demographics (Buolamwini & Buolamwini, 2018), but this is the first systematic study of politeness as a prompt variable. The research opens new directions: understanding whether politeness effects are bugs (model brittleness) or features (learned social reasoning), measuring cross-cultural politeness norms in model behavior, and designing robust systems that perform consistently regardless of user tone.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::