POEMetric: The Last Stanza of Humanity
| Authors | Bingru Li et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.03695 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large Language Models (LLMs) can compose poetry, but how far are they from human poets? In this paper, we introduce POEMetric, the first comprehensive framework for poetry evaluation, examining 1) basic instruction-following abilities in generating poems according to a certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship. We curated a human poem dataset - 203 English poems of 7 fixed forms annotated with meter, rhyme patterns and themes - and experimented with 30 LLMs for poetry generation based on the same forms and themes of the human data, totaling 6,090 LLM poems. Based on POEMetric, we assessed the performance of both human poets and LLMs through rule-based evaluation and LLM-as-a-judge, whose results were validated by human experts. Results show that, though the top model achieved high form accuracy (4.26 out of 5.00, with Gemini-2.5-Pro as a judge; same below) and theme alignment (4.99), all models failed to reach the same level of advanced abilities as human poets, who achieved unparalleled creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), and skillful use of imagery (4.49) and literary devices (4.67). Humans also defeated the best-performing LLM in overall poem quality (4.22 vs. 3.20). As such, poetry generation remains a formidable challenge for LLMs. Data and codes are released at https://github.com/Bingru-Li/POEMetric.
Engineering Breakdown
Plain English
POEMetric is the first systematic evaluation framework for assessing how well large language models can write poetry compared to human poets. The authors built a curated dataset of 203 human poems across 7 fixed forms (like sonnets, haikus) with detailed annotations for meter, rhyme, and theme, then generated 6,090 poems from 30 different LLMs using the same constraints. The framework evaluates three layers of ability: basic instruction-following (can the model follow formal constraints), advanced creative abilities (lexical diversity, emotional resonance, literary devices), and overall quality assessment including authorship attribution. This work establishes the first comprehensive benchmark for measuring the gap between LLM poetry and human composition.
Core Technical Contribution
POEMetric introduces a three-tier evaluation methodology that goes beyond traditional NLP metrics by decomposing poetry quality into measurable linguistic and stylistic components. The core innovation is the structured annotation scheme that captures formal constraints (meter, rhyme patterns), semantic properties (emotional resonance, imagery), and subjective qualities (overall quality, stylistic idiosyncrasy) in a single comparable framework. Unlike prior work that relies on single-metric evaluation or human judgment alone, POEMetric provides a layered assessment hierarchy: form compliance → creative sophistication → holistic quality. This enables fine-grained analysis of where LLMs succeed and fail in poetic composition, making it possible to systematically benchmark progress across 30 different models.
How It Works
The evaluation pipeline starts with input specifications: a poetic form (e.g., sonnet with 14 lines, ABAB CDCD EFEF GG rhyme scheme) and a theme. The framework then measures three distinct output dimensions: (1) Basic instruction-following layer evaluates whether generated poems conform to meter, rhyme pattern, line count, and thematic relevance using pattern matching and structural validation; (2) Advanced ability layer scores creativity via metrics like type-token ratio for lexical diversity, semantic similarity analysis for imagery coherence, and learned representations for emotional evocation and literary device usage; (3) General appraisal layer performs holistic quality rating and authorship classification (human vs. LLM) to assess distinguishability. The evaluation combines automated metrics with structured human annotation across all 6,090 poems, aggregating scores into a composite POEMetric score that reflects overall poetic capability. This multi-stage pipeline allows isolating which layer causes failure when an LLM-generated poem underperforms.
Production Impact
For teams building LLM-based creative writing systems, POEMetric provides a reusable evaluation framework that could replace ad-hoc human review in development cycles, potentially reducing the cost of iterative poetry or formal verse generation products. The three-tier structure lets you diagnose exactly where your model fails—whether it's basic constraint satisfaction, lack of creative sophistication, or overall quality—which directly guides fine-tuning strategies or architectural changes. Adopting this approach would require: (1) curating or licensing the 203-poem benchmark dataset with annotations, (2) implementing automated scoring functions for meter/rhyme/form compliance, and (3) running inference on your candidate models against all test cases, which scales linearly with model count but is compute-efficient since poetry is typically short text. The framework is particularly valuable for production systems in poetry generation, lyric writing, or any form-constrained creative generation where current LLMs show mixed results; however, it requires significant upfront annotation effort (the authors manually labeled all poems) and assumes forms amenable to structural validation.
Limitations and When Not to Use This
POEMetric is limited to English poetry and the 7 fixed forms represented in the 203-poem dataset—extending to other languages or free verse would require substantial retraining of the annotation scheme and likely new human evaluation. The framework assumes formal poetic structure is measurable and scoreable (meter, rhyme, line count), which works well for classical Western forms but struggles with modern experimental poetry or oral traditions that prioritize sound and performance over written form. The human authorship classification layer may become unreliable as LLMs improve, potentially making the benchmark obsolete for future model generations. Additionally, the evaluation combines automated metrics with human judgment, introducing annotation cost and potential subjectivity in assessing creativity, emotional resonance, and literary device usage—making it difficult to run real-time evaluation in production without human-in-the-loop overhead. The paper also doesn't address how POEMetric generalizes to longer poems, multiple stanzas, or narrative verse beyond the tested forms.
Research Context
This work builds on decades of computational poetry and automatic evaluation research, but represents the first attempt to create a comprehensive, multi-dimensional benchmark specifically for LLM poetry generation. It extends prior work in neural text generation evaluation (which typically use BLEU, ROUGE, or BERTScore) by introducing domain-specific metrics tied to poetic form and aesthetics rather than surface-level similarity to references. The paper sits at the intersection of generative AI capability assessment and digital humanities, filling a gap between general-purpose LLM benchmarks (like MMLU or HumanEval) and specialized evaluation frameworks like those used in machine translation or summarization. POEMetric opens the research direction of capability-aware evaluation frameworks, where assessment tiers match the complexity hierarchy of creative tasks—a pattern that could inspire future work in music generation, visual art critique, or narrative evaluation where form and aesthetics matter equally.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::