CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility
| Authors | João Silva et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.12872 |
| Download | |
| Categories | cs.CL |
Abstract
This paper reports on the development of a leaderboard of Open Large Language Models (LLM) for European Portuguese (PT-PT), and on its associated benchmarks. This leaderboard comes as a way to address a gap in the evaluation of LLM for European Portuguese, which so far had no leaderboard dedicated to this variant of the language. The paper also reports on novel benchmarks, including some that address aspects of performance that so far have not been available in benchmarks for European Portuguese, namely model safeguards and alignment to Portuguese culture. The leaderboard is available at https://huggingface.co/spaces/PORTULAN/portuguese-llm-leaderboard.
Engineering Breakdown
Plain English
This paper introduces the first dedicated leaderboard and benchmark suite for evaluating open-source large language models specifically on European Portuguese (PT-PT), filling a gap in regional language model evaluation. The researchers developed novel benchmarks that measure not just general language capabilities but also model safety mechanisms and cultural alignment to Portuguese contexts—evaluation dimensions that didn't previously exist for this language variant. The leaderboard is now live on Hugging Face and provides the community with standardized tools to measure and compare LLM performance on European Portuguese across multiple dimensions. This addresses a real problem: without language-specific evaluation infrastructure, developers building Portuguese applications had no clear way to assess which models performed best for their use cases.
Core Technical Contribution
The core novelty is the creation of a comprehensive evaluation framework tailored to European Portuguese that goes beyond standard language benchmarks. Rather than just translating existing multilingual leaderboards, the authors identified and implemented benchmarks that capture language-specific characteristics and cultural contexts unique to Portuguese speakers in Europe. The framework explicitly includes safeguard evaluation (measuring alignment and safety properties) and cultural alignment assessments—dimensions typically absent from regional language benchmarks. This represents a shift from generic multilingual evaluation toward region-aware, culturally-grounded benchmarking that recognizes Portuguese as a distinct evaluation domain rather than a minor variant of broader Romance language evaluation.
How It Works
The leaderboard operates as a modular evaluation system hosted on Hugging Face Spaces where open-source LLMs can be submitted and automatically evaluated against the curated benchmark suite. The input is a model checkpoint (typically from HuggingFace Model Hub); the system then runs the model through multiple benchmark tasks, collecting outputs for scoring. The benchmarks fall into three categories: standard language understanding and generation tasks (measuring baseline linguistic competence), safeguard evaluations (measuring whether models refuse harmful requests appropriately and align with safety principles), and cultural alignment tasks (measuring whether responses respect Portuguese cultural contexts and values). Scores are aggregated and displayed on a leaderboard ranked by multiple metrics, allowing users to sort by different evaluation dimensions depending on their application needs. The infrastructure handles inference, metric computation, and result tracking automatically, enabling continuous evaluation as new models are submitted.
Production Impact
For teams building Portuguese applications, this leaderboard eliminates guesswork when selecting base models—you can now make data-driven decisions about which open models perform best for your specific use case rather than defaulting to English-trained models or larger multilingual variants that may underperform on Portuguese. The inclusion of safety and cultural alignment benchmarks is particularly important for production systems where model behavior must match regional values and safety standards; you can now quantitatively assess whether a model exhibits problematic behavior before deployment. The standardized evaluation infrastructure reduces engineering overhead: instead of building custom evaluation harnesses for Portuguese support, teams can rely on the published benchmarks and reuse evaluation logic. However, the trade-off is that leaderboard performance doesn't guarantee production performance—task-specific fine-tuning or RLHF on your actual use case data may still be necessary, and the benchmarks may not capture all edge cases relevant to your particular application domain.
Limitations and When Not to Use This
The paper focuses on evaluation infrastructure but doesn't address model training or fine-tuning specifically for Portuguese, so organizations may still need to invest in adaptation work even after identifying the best base model on the leaderboard. The benchmarks, while comprehensive, are static snapshots—they may miss emerging failure modes or cultural shifts that occur after publication, requiring periodic re-evaluation and benchmark updates. The evaluation is limited to open-source models; proprietary systems from major providers aren't included, so comparisons between open and closed models aren't available. Additionally, the paper doesn't deeply explore how well benchmarks correlate with real-world production performance—high leaderboard scores don't guarantee good performance on your specific business metrics, and downstream task performance may require additional validation.
Research Context
This work builds on the broader movement toward language-specific and region-specific LLM evaluation, extending frameworks like MTBE (Multilingual Benchmarking and Evaluation) to focus on underrepresented language variants. It addresses a documented gap in the evaluation landscape where most leaderboards prioritize high-resource languages (English, Chinese, Spanish) with minimal coverage of regional variants and minority languages. The paper contributes to the emerging research direction of culturally-aware AI evaluation, recognizing that generic benchmarks may not capture important aspects of model behavior for specific communities. This work also enables future research on Portuguese-specific model adaptation, as it provides a standard reference point for measuring progress on language-specific capabilities that the research community can build upon.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::