On Semiotic-Grounded Interpretive Evaluation of Generative Art
| Authors | Ruixiang Jiang & Changwen Chen |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2604.08641 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Interpretation is essential to deciphering the language of art: audiences communicate with artists by recovering meaning from visual artifacts. However, current Generative Art (GenArt) evaluators remain fixated on surface-level image quality or literal prompt adherence, failing to assess the deeper symbolic or abstract meaning intended by the creator. We address this gap by formalizing a Peircean computational semiotic theory that models Human-GenArt Interaction (HGI) as cascaded semiosis. This framework reveals that artistic meaning is conveyed through three modes - iconic, symbolic, and indexical - yet existing evaluators operate heavily within the iconic mode, remaining structurally blind to the latter two. To overcome this structural blindness, we propose SemJudge. This evaluator explicitly assesses symbolic and indexical meaning in HGI via a Hierarchical Semiosis Graph (HSG) that reconstructs the meaning-making process from prompt to generated artifact. Extensive quantitative experiments show that SemJudge aligns more closely with human judgments than prior evaluators on an interpretation-intensive fine-art benchmark. User studies further demonstrate that SemJudge produces deeper, more insightful artistic interpretations, thereby paving the way for GenArt to move beyond the generation of "pretty" images toward a medium capable of expressing complex human experience. Project page: https://github.com/songrise/SemJudge.
Engineering Breakdown
Plain English
This paper addresses a critical gap in how generative art systems are evaluated. Current evaluators focus only on surface-level image quality and literal prompt matching, but fail to capture the deeper symbolic and abstract meaning that artists intend to convey. The authors propose SemJudge, an evaluation framework grounded in Peircean semiotic theory that models human-generative art interaction as a cascaded semiosis process. The framework identifies three distinct modes of meaning-making—iconic, indexical, and symbolic—and shows that existing evaluators are structurally blind to the latter two, operating almost exclusively in the iconic (visual similarity) domain.
Core Technical Contribution
The core novelty is formalizing a computational semiotic theory for evaluating generative art that moves beyond pixel-level metrics to semantic meaning assessment. The authors introduce SemJudge, which explicitly models the three Peircean sign modes (iconic, symbolic, and indexical) and constructs evaluation pipelines that can detect and score meaning across all three channels rather than only visual fidelity. This is fundamentally different from existing evaluators like CLIP-based metrics or aesthetic scorers, which collapse all meaning into a single iconic (image-to-prompt) alignment dimension. The contribution is both theoretical—formalizing semiotic grounding in computational terms—and practical, offering a concrete evaluator architecture that can operate across multiple semantic modes simultaneously.
How It Works
The system models human-generative art interaction as cascaded semiosis, where each generation step produces signs that carry meaning through three channels. The iconic mode captures visual similarity between image and referent (what CLIP-based evaluators traditionally measure). The indexical mode detects causal or contextual relationships—for example, smoke as an index of fire, or visual markers that point to external references or artist intent. The symbolic mode captures abstract associations and cultural meanings encoded in the artwork—symbols, metaphor, and conceptual layers that have no direct visual correspondence. SemJudge takes as input a generated image, the artist's prompt or intent description, and optionally contextual metadata, then routes the analysis through specialized sub-evaluators for each semiotic mode. Each mode produces a score or semantic vector; these are aggregated into a holistic judgment that reflects the artwork's success at conveying intended meaning across all three dimensions rather than just visual fidelity.
Production Impact
For production generative art systems and galleries, adopting SemJudge would shift evaluation from purely aesthetic/technical metrics to semantic coherence scoring, enabling curation of artworks by conceptual depth rather than visual quality alone. This is particularly valuable in creative platforms (like Midjourney or Runway for artists) where users care about whether their artistic vision was understood, not just pixel-perfect prompt adherence. The trade-off is computational complexity: semiotic evaluation requires language models for intent parsing, vision transformers for visual analysis, and external knowledge bases or fine-tuned symbolic reasoners for meaning extraction—likely 2-5x more inference cost than CLIP scoring. Integration into existing pipelines requires: (1) enriching prompts with artist intent metadata, (2) defining ground-truth semiotic annotations for fine-tuning mode-specific scorers, and (3) establishing domain-specific symbol dictionaries. The approach shines when generating conceptual or abstract art where traditional metrics fail; it may over-complicate evaluation of photorealistic generation where iconic alignment is the primary concern.
Limitations and When Not to Use This
The paper does not address how to handle culturally-specific or historically-evolving symbol meanings—semiotic evaluation requires substantial labeled data and domain expertise to ground symbol interpretation, which limits scalability across diverse cultural contexts. The cascaded semiosis model assumes clean separation between iconic, indexical, and symbolic modes, but these overlap heavily in real artworks, and the paper doesn't detail how mode conflicts are resolved or weighted. Ground-truth annotation of artistic intent and symbolic meaning is subjective and expensive, so the approach requires either large curated datasets or expensive human raters in the loop, creating a data bottleneck. The framework also assumes artists are operating within a rational semiotic framework; surrealist or deliberately anti-semantic art, where the artist explicitly rejects meaning, may be misjudged by a system optimizing for semantic coherence.
Research Context
This work builds on decades of semiotics theory (Peirce, 1931) and recent efforts to ground AI interpretation in formal sign theory, extending prior work on visual semantics (Antol et al. on VQA, Karpukhin et al. on dense passage retrieval) into the art evaluation domain. It addresses shortcomings of existing generative art evaluators—CLIP-based metrics (Heyes et al., 2021), aesthetic scorers (Murray et al.), and prompt-adherence measures—which are known to correlate poorly with human judgments of artistic success. The research opens a new evaluation benchmark direction: semiotic coherence scoring as a distinct task from photorealism or prompt fidelity, potentially enabling new datasets (curated GenArt collections with artist intent annotations) and spurring development of symbolic reasoning modules for art AI. This positions generative art evaluation as a semantic task rather than a purely vision task, aligning with broader trends toward multimodal understanding in AI.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::