SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attribution
| Authors | Xiachong Feng et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.18982 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Social intelligence, the ability to navigate complex interpersonal interactions, presents a fundamental challenge for language agents. Training such agents via reinforcement learning requires solving the credit assignment problem: determining how individual utterances contribute to multi-turn dialogue outcomes. Existing approaches directly employ language models to distribute episode-level rewards, yielding attributions that are retrospective and lack theoretical grounding. We propose SAVOIR (ShApley Value fOr SocIal RL), a novel principled framework grounded in cooperative game theory. Our approach combines two complementary principles: expected utility shifts evaluation from retrospective attribution to prospective valuation, capturing an utterance's strategic potential for enabling favorable future trajectories; Shapley values ensure fair credit distribution with axiomatic guarantees of efficiency, symmetry, and marginality. Experiments on the SOTOPIA benchmark demonstrate that SAVOIR achieves new state-of-the-art performance across all evaluation settings, with our 7B model matching or exceeding proprietary models including GPT-4o and Claude-3.5-Sonnet. Notably, even large reasoning models consistently underperform, suggesting social intelligence requires qualitatively different capabilities than analytical reasoning.
Engineering Breakdown
Plain English
SAVOIR addresses a critical problem in training language agents to handle complex social interactions: how to correctly assign credit for dialogue outcomes across multiple turns when using reinforcement learning. The paper proposes a framework grounded in cooperative game theory that uses Shapley values to determine which utterances actually contributed to successful social outcomes, rather than relying on retrospective language model judgments. Unlike existing approaches that look backward to explain what happened, SAVOIR evaluates each utterance's prospective strategic value—its potential to enable favorable future interactions. This principled approach provides theoretically justified reward attribution for multi-turn dialogue training.
Core Technical Contribution
The core innovation is applying Shapley value theory from cooperative game theory to solve the credit assignment problem in social dialogue RL, replacing ad-hoc language model-based attribution with a principled game-theoretic framework. SAVOIR combines two complementary evaluation principles: expected utility shifts the focus from retrospective explanation (which utterances led to success?) to prospective valuation (which utterances strategically enabled future success?). This dual mechanism grounds attribution in formal theory rather than model intuition, making the credit assignment process interpretable and reproducible. The framework is specifically designed for multi-turn interactions where individual utterances have indirect, delayed effects on outcomes that previous approaches struggled to capture correctly.
How It Works
SAVOIR operates on dialogue episodes by treating them as coalitional games where each utterance is a 'player' and the final social outcome is the payoff. For each utterance, the system computes its Shapley value—the average marginal contribution of that utterance when it's added to all possible subsets of other utterances in the dialogue. The expected utility shift principle evaluates each utterance not by looking back at what already happened, but forward by computing the expected value of game states that become possible after that utterance is made. The framework feeds these Shapley-based attributions back into the RL training loop, where each utterance receives a reward proportional to its computed Shapley value. This creates a mathematically grounded credit assignment mechanism where utterances that enable future social success (even if not immediately rewarded) are properly recognized, and the RL agent learns to optimize for long-horizon social intelligence rather than short-term reward signals.
Production Impact
For engineers building social dialogue systems, SAVOIR enables more accurate training of language agents on complex multi-turn interactions by solving a fundamental problem: correctly identifying which model generations actually matter for task success. The approach would replace heuristic reward modeling with theoretically grounded attribution, reducing the engineering effort needed to hand-craft reward functions and debug why agents learn undesirable dialogue patterns. Concretely, this means you could train conversational agents on tasks like negotiation, customer service, or collaborative problem-solving with better convergence and more interpretable learning signals. The trade-off is computational cost: computing Shapley values requires evaluating marginal contributions across dialogue combinations, which scales exponentially with episode length, so you'd need to apply it to shorter dialogue segments or use approximation techniques (sampling subsets rather than computing all) in practice. Integration would require building a Shapley attribution layer into your RL pipeline and careful validation that the prospective valuation principle captures your specific social success metrics.
Limitations and When Not to Use This
The Shapley value computation is computationally expensive—it requires evaluating 2^n possible coalitions where n is the number of utterances, making exact computation infeasible for long dialogue episodes without aggressive approximation. The paper assumes that 'social success' can be cleanly defined as a payoff function, but real social interactions involve subjective, contextual, and culturally-dependent notions of success that may not map cleanly to utility scores; this could lead to training agents optimized for the wrong definition of socially intelligent behavior. The expected utility shift principle, while theoretically sound, relies on accurate evaluation of counterfactual game states (what would happen if we added this utterance?), which requires a reliable way to simulate dialogue continuations—a non-trivial problem when using language models with inherent stochasticity. The framework has only been evaluated in controlled dialogue settings and it's unclear how well it generalizes to open-ended, real-world social interactions with multiple parties, competing objectives, or adversarial participants.
Research Context
This work builds on decades of cooperative game theory research, specifically the Shapley value framework from Shapley (1953), and applies it to a modern problem: credit assignment in deep RL for language agents. It extends prior work in RLHF (Reinforcement Learning from Human Feedback) and dialogue RL by replacing heuristic or model-based reward attribution with formal game-theoretic principles. The paper contributes to the broader research direction of interpretable and theoretically grounded reward modeling in language agent training, which has become critical as RL is applied to increasingly complex interactive tasks. It opens the door to applying other cooperative game theory concepts (core solutions, nucleolus, etc.) to RL credit assignment problems, potentially inspiring similar principled approaches in other domains like multi-agent coordination or hierarchical policy learning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::