StyleID: A Perception-Aware Dataset and Metric for Stylization-Agnostic Facial Identity Recognition
| Authors | Kwan Yun et al. |
| Year | 2026 |
| HF Upvotes | 23 |
| arXiv | 2604.21689 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Creative face stylization aims to render portraits in diverse visual idioms such as cartoons, sketches, and paintings while retaining recognizable identity. However, current identity encoders, which are typically trained and calibrated on natural photographs, exhibit severe brittleness under stylization. They often mistake changes in texture or color palette for identity drift or fail to detect geometric exaggerations. This reveals the lack of a style-agnostic framework to evaluate and supervise identity consistency across varying styles and strengths. To address this gap, we introduce StyleID, a human perception-aware dataset and evaluation framework for facial identity under stylization. StyleID comprises two datasets: (i) StyleBench-H, a benchmark that captures human same-different verification judgments across diffusion- and flow-matching-based stylization at multiple style strengths, and (ii) StyleBench-S, a supervision set derived from psychometric recognition-strength curves obtained through controlled two-alternative forced-choice (2AFC) experiments. Leveraging StyleBench-S, we fine-tune existing semantic encoders to align their similarity orderings with human perception across styles and strengths. Experiments demonstrate that our calibrated models yield significantly higher correlation with human judgments and enhanced robustness for out-of-domain, artist drawn portraits. All of our datasets, code, and pretrained models are publicly available at https://kwanyun.github.io/StyleID_page/
Engineering Breakdown
Plain English
This paper addresses a critical failure mode in facial identity recognition systems: current identity encoders trained on natural photographs completely break down when faces are stylized (cartoons, sketches, paintings). The authors introduce StyleID, a human perception-aware dataset and evaluation framework that measures identity consistency across different artistic styles and stylization strengths. The key insight is that style-agnostic identity encoding has never been properly benchmarked or evaluated before, leaving a gap between what practitioners think their systems can do and what they actually do under stylization. By providing StyleBench-H (a human-calibrated benchmark capturing same-different verification judgments), the paper gives the community its first principled way to measure and improve identity preservation across style variations.
Core Technical Contribution
The core novelty is introducing a human perception-calibrated evaluation framework for facial identity under stylization—something that didn't exist before in the literature. Rather than assuming that identity encoders trained on natural images will generalize to stylized faces, the authors explicitly measure the gap and provide StyleBench-H as ground truth based on human judgments of identity consistency across styles. This flips the typical approach: instead of building a new encoder architecture, they're building the evaluation infrastructure and benchmark dataset that will allow the community to properly measure and supervise identity preservation. The human-perception alignment is critical—it captures the insight that identity recognition under stylization is fundamentally a perceptual problem, not just a technical metric optimization problem.
How It Works
StyleID operates in two phases: data collection and evaluation framework construction. In the data collection phase, the authors curate StyleBench-H by taking portrait images, applying stylization transformations (cartoon, sketch, painting, etc.) at varying strengths, and collecting human annotations on whether pairs of stylized faces are perceived as the same identity or different identities. This creates a ground-truth dataset where human perception is the source of truth, not algorithmic confidence scores. The evaluation framework then uses StyleBench-H to benchmark existing identity encoders and measure their brittleness under stylization—capturing where they confuse style changes for identity changes or fail to recognize geometric exaggerations that humans easily see through. The framework likely computes metrics like verification accuracy, ROC curves, and style-agnostic embedding quality by comparing encoder outputs against human judgments, enabling practitioners to identify which components of their pipeline fail under stylization.
Production Impact
For teams building face recognition systems that need to work on user-generated content (where stylized, filtered, and edited faces are common), StyleID provides the first principled way to measure and debug identity preservation failures. Instead of discovering at deployment that your identity encoder fails on cartoon profile pictures or anime artwork, you can now benchmark against StyleBench-H and explicitly supervise for style-agnostic features during training. This directly improves production reliability for applications like content moderation, user verification on social platforms, and identity-based search where users often upload stylized or artistic versions of their photos. The trade-off is data collection overhead—you need human annotations for stylized face pairs at multiple stylization strengths—and potential retraining of identity encoders to be style-agnostic, which increases model complexity and may require larger embedding dimensions. Integration is straightforward: use StyleBench-H as a validation set and potentially as a contrastive training signal to regularize encoders against style brittleness.
Limitations and When Not to Use This
The paper assumes that human perception of identity under stylization is stable and well-defined, but artistic style can be subjective—different humans might disagree on whether a heavily exaggerated cartoon maintains identity. StyleBench-H is a snapshot of identity perception and may not generalize to novel stylization techniques or extreme transformations beyond what was included in data collection. The framework also doesn't address the root cause of encoder brittleness (e.g., why do standard encoders use style-dependent features?), so practitioners still need to either retrain encoders from scratch or apply post-hoc fine-tuning, both expensive. Finally, the paper evaluates identity consistency but doesn't tackle the geometric and semantic understanding required to preserve identity under extreme exaggeration or viewpoint changes that stylization often introduces—a stylized face at a different angle is a fundamentally different recognition problem.
Research Context
This work builds on decades of face recognition research (FaceNet, ArcFace, etc.) but identifies a previously unquantified gap: these models are benchmarked on natural image distributions (LFW, VGGFace2) and fail catastrophically on out-of-distribution stylized faces. It joins recent work on out-of-distribution robustness in vision, but focuses specifically on the artistic stylization domain where style variations are intentional rather than accidental. The paper contributes to the broader evaluation infrastructure challenge in ML—similar to how MNIST broke when real handwriting was introduced, or how ImageNet didn't capture distribution shifts; StyleID recognizes that identity encoders need human-aligned benchmarks for realistic deployment. This opens up a research direction on style-agnostic representation learning and could inspire similar human-calibrated frameworks for other perceptual tasks (emotion recognition, age estimation) under stylization.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::