Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
| Authors | NVIDIA et al. |
| Year | 2026 |
| HF Upvotes | 33 |
| arXiv | 2604.12374 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemotron 3 Super is the first model in the Nemotron 3 family to 1) be pre-trained in NVFP4, 2) leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both accuracy per FLOP and accuracy per parameter, and 3) include MTP layers for inference acceleration through native speculative decoding. We pre-trained Nemotron 3 Super on 25 trillion tokens followed by post-training using supervised fine tuning (SFT) and reinforcement learning (RL). The final model supports up to 1M context length and achieves comparable accuracy on common benchmarks, while also achieving up to 2.2x and 7.5x higher inference throughput compared to GPT-OSS-120B and Qwen3.5-122B, respectively. Nemotron 3 Super datasets, along with the base, post-trained, and quantized checkpoints, are open-sourced on HuggingFace.
Engineering Breakdown
Plain English
NVIDIA's Nemotron 3 Super is a 120 billion parameter model that uses a hybrid Mamba-Attention Mixture-of-Experts design with only 12 billion active parameters during inference, trained on 25 trillion tokens using a novel NVFP4 quantization format from the start of pre-training. The model incorporates LatentMoE, a new MoE architecture optimizing for both accuracy-per-FLOP and accuracy-per-parameter efficiency, plus MTP layers enabling native speculative decoding for faster inference. The final system achieves up to 2.2x and 7.5x higher inference throughput compared to baselines while maintaining competitive accuracy on standard benchmarks and supporting 1 million token context windows.
Core Technical Contribution
The paper introduces three novel technical contributions: (1) NVFP4 pre-training, a low-precision floating-point format that enables dense model training without the typical accuracy degradation of ultra-low-bit quantization, (2) LatentMoE, a Mixture-of-Experts architecture that jointly optimizes computational efficiency and parameter efficiency by reducing the active parameter count while maintaining per-operation accuracy, and (3) MTP (Multi-Token Prediction) layers that natively support speculative decoding without requiring external draft models. The core innovation is demonstrating that hybrid Mamba-Attention architectures combined with expert routing can achieve both parameter and compute efficiency simultaneously, a problem that prior MoE work largely treated as a trade-off rather than a jointly solvable optimization.
How It Works
Nemotron 3 Super operates as a hybrid architecture that selectively uses either Mamba (a state-space model with linear complexity) or Attention mechanisms depending on the token position and context, combined with a learned gating network that routes tokens to sparse experts. During pre-training, the entire model (including quantized weights) is trained end-to-end on NVFP4, a custom low-precision format that represents weights with fewer bits while preserving gradient flow and convergence properties. The LatentMoE component uses a learned routing mechanism that selects which expert subset processes each token, reducing active parameters from 120B to 12B while the model learns which routing decisions maintain accuracy. For inference, MTP layers predict multiple future tokens in parallel, and these predictions feed into a speculative decoding mechanism that allows the model to verify and accept multiple tokens per forward pass rather than one token per pass, effectively amortizing the cost of expensive attention computations across multiple predictions.
Production Impact
For engineers deploying large language models in production, this work directly translates to 2-7x higher token throughput at equivalent accuracy, which substantially reduces serving costs and latency for latency-sensitive applications like real-time chat or code completion. The low active parameter count (12B out of 120B) enables running this model on smaller GPUs and in more bandwidth-constrained environments than a fully-dense 120B model, making enterprise deployment feasible without requiring the latest A100/H100 clusters. The NVFP4 pre-training approach is particularly valuable because it eliminates the post-training quantization step — you don't need to train a dense model then compress it, which saves weeks of training and simplifies the pipeline. However, the trade-off is that you must adopt NVFP4 format support in your inference stack (likely requires custom CUDA kernels or vendor support), and the hybrid Mamba-Attention routing adds complexity compared to pure-Attention models, requiring careful tuning of the routing mechanism for your specific workload.
Limitations and When Not to Use This
The paper does not address robustness of the routing mechanism to adversarial inputs or distribution shift — if the token patterns in production differ significantly from pre-training, the learned expert routing may degrade gracefully or collapse entirely. NVFP4 is a proprietary NVIDIA format, which means this approach is not reproducible or adoptable by teams lacking NVIDIA hardware partnerships; the generalizability to other custom quantization formats is unclear. The 1M context length claim is mentioned but no detailed benchmarks or memory consumption analysis is provided for extreme context scenarios, making it unclear if the model maintains efficiency at these scales or if it falls back to approximate attention. Additionally, the paper does not provide ablations separating the contribution of LatentMoE versus MTP versus NVFP4, so it's unclear which component drives the majority of the 2-7x speedup, limiting architectural guidance for practitioners building their own systems.
Research Context
This work builds on several threads: the rise of Mixture-of-Experts sparse models (pioneered by Switch Transformers and GShard) that trade parameter count for compute efficiency, recent work on Mamba and state-space models as alternatives to full attention, and the quantization-aware training direction that trains models in low precision from the beginning rather than post-hoc. It sits at the intersection of two efficiency goals — parameter efficiency (fewer active parameters) and compute efficiency (fewer FLOPs) — which prior work often treated separately; Nemotron 3 Super demonstrates that hybrid architectures can address both simultaneously. The work likely advances the baseline on standard LLM benchmarks (MMLU, HumanEval, etc.) while claiming efficiency gains, positioning it as a new efficiency frontier in the large language model arms race where compute-optimal scaling laws drive model design.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::