Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs
| Authors | Samuel G. Balter et al. |
| Year | 2026 |
| HF Upvotes | 6 |
| arXiv | 2604.18203 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Multimodal LLMs can accurately perceive numerical content across modalities yet fail to perform exact multi-digit multiplication when the identical underlying arithmetic problem is presented as numerals, number words, images, or in audio form. Because existing benchmarks often lack systematically paired instances across modalities, it remains difficult to compare genuine arithmetic limits within and across model families. We therefore introduce a controlled multimodal multiplication benchmark that factorially varies digit length, digit sparsity, representation (e.g., numerals vs. number words), and modality (text, rendered images, audio), with paired instances from a reproducible generator. We also define arithmetic load, C, as the product of the total and non-zero digit count as a compact, mechanistically motivated proxy for operation count. Across evaluations, accuracy falls sharply as C grows, often nearing zero by C > 100. Indeed, C remains predictive of performance across modalities and models, with R-squared often > 0.5, nearing the value from more complex measures of arithmetic load that count the number of intermediate arithmetic steps. A separate perception-versus-computation decomposition shows that multimodal degradation is primarily computational rather than perceptual: on matched-perception checks, models are near-perfect (> 99%) across modalities, even when multiplication accuracy drops. Beyond measuring when models fail, we ask which procedures they are predisposed to follow. We introduce a forced-completion loss probe that scores heuristic-specific reasoning prefixes--including columnar multiplication, distributive decomposition, and rounding/compensation. Here, decomposition is favored in both text and vision modalities; heuristic-specific LoRA adapters produce near-orthogonal updates yet degrade accuracy, indicating the base model maintains a well-tuned internal router.
Engineering Breakdown
Plain English
This paper identifies a critical failure mode in multimodal large language models: they can perceive numerical content across different modalities (text, images, audio) but consistently fail at exact multi-digit multiplication when the same arithmetic problem is presented in different formats. The authors built a controlled benchmark called a multimodal multiplication benchmark that systematically varies problem difficulty (digit length, sparsity), representation format (numerals vs. number words), and modality (text, images, audio) to isolate where models actually break down. They introduce a metric called arithmetic load (C) to quantify problem complexity as the product of total and non-zero digit counts. This structured approach reveals genuine arithmetic limitations that existing fragmented benchmarks couldn't detect because they lack paired instances across modalities.
Core Technical Contribution
The core contribution is a factorially designed controlled benchmark that decouples representation format and modality from arithmetic complexity itself. Unlike prior work that tested arithmetic on scattered datasets without systematic pairing, this benchmark generates reproducible problem instances where you can directly compare a model's performance on the exact same multiplication task rendered as numerals, spelled-out words, images, or audio. The arithmetic load metric (C) provides a quantitative framework for measuring problem difficulty independent of presentation format, making it possible to distinguish between modality-specific failures and genuine mathematical reasoning failures. This enables fair cross-modal comparison and isolates whether a model's arithmetic errors stem from perception issues or computational/reasoning limitations.
How It Works
The benchmark operates as a controlled factor experiment with four dimensions: digit length (how many digits in the multiplicands), digit sparsity (proportion of non-zero digits), representation format (e.g., '42' as numeral vs. 'forty-two' as words), and modality (text, rendered image, or audio). For each combination of these factors, the system generates paired problem instances—the identical multiplication problem rendered across all modalities. A model receives the problem in one modality (e.g., an audio file saying 'seventeen times twenty-three'), processes it through its multimodal input encoder, and outputs the correct answer. The arithmetic load metric C is computed as total_digits × non_zero_digits, allowing researchers to plot accuracy against problem complexity while controlling all other variables. By holding everything constant except one dimension at a time, researchers can measure whether performance degradation comes from perception (modality/format handling) or computation (actual arithmetic reasoning).
Production Impact
For production systems, this work directly informs reliability assessment of multimodal AI deployed in domains requiring accurate arithmetic: financial calculations, scientific data analysis, medical dosing from charts or dictations, and e-commerce pricing across voice, visual, and text interfaces. Teams can now use this benchmark to benchmark their multimodal models and establish which problem sizes are safe for deployment—if a model fails on all 4-digit multiplications regardless of format, you know your system cannot reliably handle financial transactions above $9999. The arithmetic load metric provides a quantitative threshold you can use to gate features: only enable voice-based math assistance for multiplication below C=20, for instance. However, this requires instrumenting your inference pipeline to detect when a problem exceeds safe complexity bounds, adding runtime classification overhead. The benchmark's factorial design means validating a new model architecture requires testing across dozens of format/modality combinations, increasing evaluation cost significantly but catching modality-specific regressions early.
Limitations and When Not to Use This
The paper focuses narrowly on multiplication and does not test whether findings generalize to other arithmetic operations (division, square roots) or to more complex reasoning tasks (multi-step problems, word problems with irrelevant information). The controlled benchmark itself is synthetic and may not reflect the distribution of arithmetic problems encountered in real-world multimodal data, where formatting, audio quality, and image resolution vary unpredictably. The arithmetic load metric assumes linear difficulty scaling (C = total × non-zero) which may not hold empirically—doubling digit count might cause superlinear accuracy drops due to attention span or numerical precision limits in transformers. The paper does not provide solutions or training techniques to fix these failures; it characterizes the problem but leaves intervention strategies (specialized numeric modules, hard-coded calculators, separate training on arithmetic-heavy data) to future work.
Research Context
This work builds on growing recognition that standard language model benchmarks (like MMLU or GSM8K) inadequately measure arithmetic reasoning, especially in multimodal settings where perception errors conflate with computation errors. It extends prior work on mathematical reasoning in LLMs (like chain-of-thought prompting) by introducing systematic factorial benchmark design, similar to how controlled experiments have validated progress on other structured reasoning tasks. The paper addresses a gap in the multimodal LLM evaluation landscape: benchmarks like LLaVA-Bench and SEED-Bench test general visual understanding but do not isolate arithmetic failures with paired cross-modal instances. This work opens a research direction toward building specialized multimodal arithmetic modules or hybrid systems that route numerical problems to symbolic calculators rather than relying on end-to-end neural processing.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::