Current LLMs still cannot 'talk much' about grammar modules: Evidence from syntax
| Authors | Mohammed Q. Shormani |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.20114 |
| Download | |
| Categories | cs.CL |
Abstract
We aim to examine the extent to which Large Language Models (LLMs) can 'talk much' about grammar modules, providing evidence from syntax core properties translated by ChatGPT into Arabic. We collected 44 terms from generative syntax previous works, including books and journal articles, as well as from our experience in the field. These terms were translated by humans, and then by ChatGPT-5. We then analyzed and compared both translations. We used an analytical and comparative approach in our analysis. Findings unveil that LLMs still cannot 'talk much' about the core syntax properties embedded in the terms under study involving several syntactic and semantic challenges: only 25% of ChatGPT translations were accurate, while 38.6% were inaccurate, and 36.4.% were partially correct, which we consider appropriate. Based on these findings, a set of actionable strategies were proposed, the most notable of which is a close collaboration between AI specialists and linguists to better LLMs' working mechanism for accurate or at least appropriate translation.
Engineering Breakdown
Plain English
This paper evaluates how well ChatGPT-5 can translate specialized generative syntax terminology from English to Arabic, a task requiring deep understanding of linguistic concepts. The authors collected 44 core syntax terms from academic literature and compared human translations against ChatGPT's outputs using manual analysis. Results showed that LLMs struggle significantly with this domain: only 25% of ChatGPT translations were fully accurate, 38.6% were completely wrong, and 36.4% were partially correct. This demonstrates that current LLMs lack robust understanding of technical linguistic properties, even in a language generation task that should theoretically be within their capabilities.
Core Technical Contribution
The core contribution is a systematic evaluation framework measuring LLM performance on specialized technical terminology translation, specifically targeting syntax-related concepts in computational linguistics. Rather than testing general translation ability, the authors isolate a narrow, well-defined domain where translation quality directly reflects whether the model actually understands the underlying linguistic constructs versus just performing surface-level token mapping. This reveals a critical gap: LLMs can generate fluent Arabic but fail to preserve the precise semantic meaning required for academic and technical communication. The finding that 74% of translations contain errors or partial correctness suggests a fundamental limitation in how LLMs internalize and manipulate formal linguistic theory.
How It Works
The experimental pipeline starts with extracting 44 key generative syntax terms from established academic sources—books, journals, and the authors' own field expertise. These terms are then translated independently by human linguists (establishing a gold-standard reference set) and by ChatGPT-5 using identical prompts. The translations are then subjected to comparative manual analysis where evaluators categorize each ChatGPT output into three buckets: fully accurate (preserves technical meaning and structure), inaccurate (introduces errors or misses core concepts), or partially correct (captures some aspects but with significant gaps). The analysis appears to be qualitative rather than using automated metrics, examining both syntactic accuracy and semantic preservation. Results are aggregated into percentages to quantify how often the model succeeds or fails across the full terminology set.
Production Impact
For engineering teams building multilingual NLP systems or translation pipelines, this paper signals that domain-specific terminology requires additional validation layers beyond standard machine translation evaluation metrics. If your production system relies on ChatGPT for translating technical documentation, academic content, or specialized domain knowledge into low-resource languages, you cannot assume end-to-end translation quality without explicit human review—especially for linguistics, mathematics, or other formal domains where precision matters. This means implementing human-in-the-loop validation for high-stakes translations, or alternatively, investing in fine-tuning smaller models on domain-specific terminology pairs rather than relying on general-purpose LLMs. The 25% accuracy baseline suggests you'd need to budget for substantial quality assurance overhead if deploying such a system. For Arabic-specific applications, you may need to augment ChatGPT with retrieval-augmented generation (RAG) systems that ground translations in terminology databases.
Limitations and When Not to Use This
This study evaluates only ChatGPT-5 on a specific task (syntax terminology translation to Arabic), so results may not generalize to other LLMs (Claude, Gemini, Llama) or other language pairs and technical domains. The evaluation set is small (44 terms) and drawn from a single subfield of linguistics, limiting statistical power and breadth of conclusions. The paper lacks quantitative inter-annotator agreement metrics or a detailed rubric defining what constitutes 'accurate,' 'inaccurate,' or 'partially correct,' which introduces subjectivity and makes results harder to reproduce. Additionally, there's no analysis of failure modes—the paper doesn't explain why the model fails (insufficient training data in Arabic technical language, architectural limitations in representing formal syntax, or prompt engineering issues) or whether simple techniques like chain-of-thought prompting could improve performance. Follow-up work should compare against other models, test on larger terminology sets, and employ quantitative evaluation with inter-rater reliability scores.
Research Context
This work contributes to a growing body of research examining LLM limitations on formal, technical tasks—building on prior work investigating whether LLMs truly understand linguistics versus performing sophisticated pattern matching. It sits at the intersection of multilingual NLP evaluation and domain-specific translation, relevant to benchmarking studies like XLMR and BERTScore but focused on terminology-level precision rather than fluency. The findings reinforce concerns raised in recent papers questioning whether LLMs have genuine linguistic knowledge, particularly in low-resource languages like Arabic. This opens research directions into specialized fine-tuning for technical translation, multilingual terminology databases for RAG systems, and better evaluation frameworks for assessing domain knowledge preservation across language boundaries.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::