Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study
| Authors | Liel Binyamin & Elior Sulem |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.12906 |
| Download | |
| Categories | cs.CL, cs.AI |
Abstract
Research on developmentally plausible language models has largely focused on English, leaving open questions about multilingual settings. We present a systematic study of compact language models by extending BabyBERTa to English-French scenarios under strictly size-matched data conditions, covering monolingual, bilingual, and cross-lingual settings. Our design contrasts two types of training corpora: (i) child-directed speech (about 2.5M tokens), following BabyBERTa and related work, and (ii) multi-domain corpora (about 10M tokens), extending the BabyLM framework to French. To enable fair evaluation, we also introduce new resources, including French versions of QAMR and QASRL, as well as English and French multi-domain corpora. We evaluate the models on both syntactic and semantic tasks and compare them with models trained on Wikipedia-only data. The results reveal context-dependent effects: training on Wikipedia consistently benefits semantic tasks, whereas child-directed speech improves grammatical judgments in monolingual settings. Bilingual pretraining yields notable gains for textual entailment, with particularly strong improvements for French. Importantly, similar patterns emerge across BabyBERTa, RoBERTa, and LTG-BERT, suggesting consistent trends across architectures.
Engineering Breakdown
Plain English
This paper studies how compact language models learn language in multilingual settings, specifically extending BabyBERTa (a small model trained on child-directed speech) to English-French scenarios with controlled data sizes. The researchers tested three training setups: monolingual English, monolingual French, and bilingual mixed, using both child-directed speech (~2.5M tokens, following prior work) and multi-domain corpora (~10M tokens, a new extension). They created new French evaluation resources (QAMR and QASRL datasets) and benchmarked models on syntactic and semantic tasks to understand how multilingual exposure affects language learning at small scale.
Core Technical Contribution
The core novelty is a systematic empirical study of how multilingual training affects compact language models under strict size-matched conditions—something largely unexplored outside English. The authors extend the BabyLM framework (which mimics child language acquisition) to multilingual settings by introducing French child-directed speech corpora and multi-domain data matched in token count to English baselines. They also contribute new evaluation infrastructure: French versions of structured semantic role labeling tasks (QAMR/QASRL) enabling fair cross-lingual comparison. This work surfaces whether and how bilingual exposure helps or hinders learning efficiency in resource-constrained settings.
How It Works
The study trains BERT-like models (using the BabyBERTa architecture) on four distinct corpora: English child speech, French child speech, English multi-domain, and French multi-domain, each controlled to match token budgets (~2.5M or ~10M). During training, models learn masked language modeling (predicting masked tokens from context) on these limited datasets, mirroring how children learn from bounded input. For bilingual conditions, the training data mixes English and French tokens at various ratios. Post-training, models are evaluated on downstream tasks: syntactic benchmarks (likely constituency/dependency parsing or tagging) and semantic tasks (QAMR/QASRL role labeling), measuring how well the limited pretraining transfers. The researchers compare cross-lingual transfer—whether French pretraining helps English tasks or vice versa—and measure whether bilingual models outperform monolingual ones given the same total tokens.
Production Impact
For engineers building multilingual NLP systems with resource constraints (edge devices, low-bandwidth settings, or rapid deployment scenarios), this work provides empirical guidance on whether multilingual pretraining saves or wastes training compute. If bilingual training underperforms two monolingual models given the same budget, teams should separate models; if it matches or exceeds both, it justifies unified architectures. The new French datasets (QAMR/QASRL) reduce engineering effort for multilingual task evaluation—you no longer need to manually translate or create task resources from scratch. This is directly applicable to production pipelines handling multiple languages: understanding the token-efficiency trade-off of multilingual vs. monolingual models informs architecture decisions and data allocation. However, results are specific to compact models (small BERT-scale); scaling laws may differ for larger models, so you cannot assume findings transfer to production models with billions of parameters.
Limitations and When Not to Use This
The study is limited to a narrow set of language pairs (English-French) and model architectures (BERT-family only), leaving open questions about other language combinations, distant language pairs, or modern architectures like GPT-style decoders. The evaluation focuses on relatively structured syntactic and semantic tasks; it does not measure performance on generation, QA, or open-ended tasks where multilingual interference might manifest differently. The data regime is artificially small (~2.5M to ~10M tokens) and does not reflect production-scale pretraining, so findings about data efficiency may not hold when scaling to billions of tokens. Additionally, the paper does not deeply investigate failure modes: which linguistic phenomena suffer most in bilingual settings, whether certain languages act as negative transfer for others, or whether model capacity (hidden size, depth) interacts with multilingual training in unexpected ways.
Research Context
This work builds directly on BabyBERTa and the BabyLM challenge, which investigate how language models learn from child-scale data (~10-100M tokens), mimicking cognitive science questions about language acquisition. It extends the multilingual pretraining literature (prior work on mBERT, XLM-R) downward into the compact, data-scarce regime where the multilingual trade-off is most acute. The paper contributes to a growing body of work on developmentally plausible language learning, bridging NLP and cognitive modeling. By introducing French versions of structured task datasets and multi-domain corpora, it removes barriers to multilingual small-model research and enables future work on language interaction effects, cross-lingual transfer at small scale, and sample efficiency for multilingual learning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::