PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech
| Authors | Venkata Pushpak Teja Menta |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2604.25476 |
| Download | |
| Code | https://github.com/praxelhq/psp-eval |
Abstract
Standard text-to-speech (TTS) evaluation measures intelligibility (WER, CER) and overall naturalness (MOS, UTMOS) but does not quantify accent. A synthesiser may score well on all four yet sound non-native on features that are phonemic in the target language. For Indic languages, these features include retroflex articulation, aspiration, vowel length, and the Tamil retroflex approximant (letter zha). We present PSP, the Phoneme Substitution Profile, an interpretable, per-phonological-dimension accent benchmark for Indic TTS. PSP decomposes accent into six complementary dimensions: retroflex collapse rate (RR), aspiration fidelity (AF), vowel-length fidelity (LF), Tamil-zha fidelity (ZF), Frechet Audio Distance (FAD), and prosodic signature divergence (PSD). The first four are measured via forced alignment plus native-speaker-centroid acoustic probes over Wav2Vec2-XLS-R layer-9 embeddings; the latter two are corpus-level distributional distances. In this v1 we benchmark four commercial and open-source systems (ElevenLabs v3, Cartesia Sonic-3, Sarvam Bulbul, Indic Parler-TTS) on Hindi, Telugu, and Tamil pilot sets, with a fifth system (Praxy Voice) included on all three languages, plus an R5->R6 case study on Telugu. Three findings: (i) retroflex collapse grows monotonically with phonological difficulty Hindi < Telugu < Tamil (~1%, ~40%, ~68%); (ii) PSP ordering diverges from WER ordering -- commercial WER-leaders do not uniformly lead on retroflex or prosodic fidelity; (iii) no single system is Pareto-optimal across all six dimensions. We release native reference centroids (500 clips per language), 1000-clip embeddings for FAD, 500-clip prosodic feature matrices for PSD, 300-utterance golden sets per language, scoring code under MIT, and centroids under CC-BY. Formal MOS-correlation is deferred to v2; v1 reports five internal-consistency signals plus a native-audio sanity check.
Engineering Breakdown
Plain English
This paper introduces PSP (Phoneme Substitution Profile), a new evaluation benchmark specifically designed to measure accent quality in text-to-speech systems for Indic languages. Standard TTS metrics like WER, CER, MOS, and UTMOS measure intelligibility and naturalness but fail to capture accent characteristics that are phonemically distinct in Indian languages—such as retroflex articulation, aspiration, vowel length, and the Tamil retroflex approximant (zha). PSP decomposes accent into six measurable dimensions: retroflex collapse rate, aspiration fidelity, vowel-length fidelity, Tamil-zha fidelity, Frechet Audio Distance, and prosodic signature divergence. This allows TTS systems that pass traditional benchmarks but sound non-native to be identified and improved.
Core Technical Contribution
The core novelty is decomposing the abstract concept of 'accent' into six interpretable, measurable phonological dimensions specific to Indic language phonemics. Prior work treated accent as a single latent variable or ignored it entirely in favor of generic naturalness metrics. PSP operationalizes four phoneme-specific dimensions (retroflex collapse rate, aspiration fidelity, vowel-length fidelity, zha fidelity) via direct phonological analysis, plus two acoustic dimensions (FAD and PSD) to capture broader spectral and prosodic drift. This per-dimension breakdown allows engineers to pinpoint which accent features their TTS model is failing on and target fixes accordingly, rather than receiving a single opaque 'accent score'.
How It Works
PSP operates in two stages. First, for phoneme-specific dimensions: the system extracts target phonemes from ground-truth transcripts, compares synthesized audio against reference native speaker recordings, and measures substitution or collapse rates (e.g., retroflex /ṭ/ becoming alveolar /t/, or aspirated /kʰ/ becoming unaspirated /k/). Aspiration, vowel length, and zha fidelity are quantified as binary correctness or frame-level acoustic alignment metrics. Second, for acoustic dimensions: Frechet Audio Distance (FAD) computes the Fréchet distance between synthetic and natural audio embeddings in a pre-trained audio representation space (likely from a model like VGGish or wav2vec), and prosodic signature divergence (PSD) measures pitch, energy, and duration distributions across the utterance. The six scores are produced independently and can be aggregated or inspected in isolation to form a full accent profile.
Production Impact
For production TTS teams, PSP transforms accent debugging from guesswork into systematic diagnosis. Instead of hearing a TTS system sounds 'off' and having no way to fix it, engineers can run PSP, see that retroflex collapse rate is 15% and aspiration fidelity is 87%, then focus phonological training or phoneme-specific fine-tuning on those weak points. This is especially valuable for low-resource Indic language TTS, where hiring native linguists to audit every failure is expensive. The benchmark requires a set of reference native speaker recordings (for FAD and per-phoneme comparison) and access to a tokenized phoneme inventory, adding minimal computational overhead to evaluation pipelines (likely single-digit GPU minutes per utterance). Trade-offs include: requiring high-quality reference corpora for each language/dialect, needing linguistically annotated phoneme inventories upfront, and the fact that PSD and FAD are still somewhat opaque 'black box' metrics that don't point to specific fixes.
Limitations and When Not to Use This
PSP assumes that the six dimensions are sufficient to capture all accent-relevant phonology—but Indic languages have additional phonemic features (nasalization, gemination, vowel harmony in some systems) that may not be fully covered. The benchmark also requires high-quality reference recordings and accurate phoneme-level alignment, which may not exist for all Indic languages or dialects, limiting generalization. FAD and PSD rely on pre-trained embeddings (VGGish, wav2vec, etc.) trained on general audio; these may not be optimized for Indic phonological distinctions and could miss subtle accent shifts that native listeners catch. Finally, the paper does not address correlations between the six dimensions—if retroflex collapse and aspiration errors are correlated with prosodic drift, the dimensions are not truly independent, which could make diagnosis and remediation less straightforward than the framework suggests.
Research Context
This work builds on decades of accent research in speech science and the recent surge in TTS for low-resource languages, where accent is a known quality bottleneck. It extends classical phonological analysis (identifying language-specific phonemic features) into the neural TTS era, where end-to-end models can gloss over phonological detail. Prior TTS benchmarks (Common Voice, MLS) focus on ASR data collection but not TTS evaluation; BLSTM-based metrics like UTMOS measure overall quality but not accent. PSP is inspired by multilingual ASR accent evaluation work but tailored to the synthesis direction. This opens a research direction: developing language-specific phonological evaluation profiles for other language families (Sino-Tibetan, Austroasiatic, Afro-Asiatic) and investigating whether the six dimensions generalize or need adaptation.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::