LightThinker++: From Reasoning Compression to Memory Management
| Authors | Yuqi Zhu et al. |
| Year | 2026 |
| HF Upvotes | 33 |
| arXiv | 2604.03679 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large language models (LLMs) excel at complex reasoning, yet their efficiency is limited by the surging cognitive overhead of long thought traces. In this paper, we propose LightThinker, a method that enables LLMs to dynamically compress intermediate thoughts into compact semantic representations. However, static compression often struggles with complex reasoning where the irreversible loss of intermediate details can lead to logical bottlenecks. To address this, we evolve the framework into LightThinker++, introducing Explicit Adaptive Memory Management. This paradigm shifts to behavioral-level management by incorporating explicit memory primitives, supported by a specialized trajectory synthesis pipeline to train purposeful memory scheduling. Extensive experiments demonstrate the framework's versatility across three dimensions. (1) LightThinker reduces peak token usage by 70% and inference time by 26% with minimal accuracy loss. (2) In standard reasoning, LightThinker++ slashes peak token usage by 69.9% while yielding a +2.42% accuracy gain under the same context budget for maximum performance. (3) Most notably, in long-horizon agentic tasks, it maintains a stable footprint beyond 80 rounds (a 60%-70% reduction), achieving an average performance gain of 14.8% across different complex scenarios. Overall, our work provides a scalable direction for sustaining deep LLM reasoning over extended horizons with minimal overhead.
Engineering Breakdown
Plain English
This paper addresses a critical efficiency problem in large language models: when LLMs perform complex reasoning with chain-of-thought prompting, they generate long intermediate thought traces that consume significant computational resources. The authors propose LightThinker++, which dynamically compresses these intermediate thoughts into compact semantic representations while preserving reasoning quality through an explicit adaptive memory management system. The key innovation moves beyond static compression by introducing specialized memory primitives and a trajectory synthesis pipeline that learns when and how to schedule memory operations during inference. This enables LLMs to maintain logical correctness on complex reasoning tasks while reducing the cognitive overhead of storing full thought traces.
Core Technical Contribution
The core novelty is a two-stage framework that treats reasoning compression as a memory management problem rather than pure information loss. LightThinker introduces dynamic semantic compression of intermediate thoughts, while LightThinker++ adds Explicit Adaptive Memory Management (EAMM)—a behavioral-level system with primitive memory operations (read, write, forget, consolidate) that can be scheduled by the model during inference. This moves beyond prior approaches that either kept all thoughts (expensive) or discarded them irreversibly (losing logical information), instead treating memory as a learnable resource that the model actively manages. The trajectory synthesis pipeline trains the model to make purposeful memory scheduling decisions, learning which intermediate details are critical for downstream reasoning steps.
How It Works
The system operates on chain-of-thought reasoning traces, where an LLM generates step-by-step intermediate thoughts before producing a final answer. Instead of storing full token sequences, LightThinker compresses each thought into a compact semantic embedding using learned compression functions (likely attention-based pooling or learned projections). During inference, LightThinker++ adds explicit memory primitives: the model can write compressed thoughts to memory, read previously stored thoughts, schedule forgetting of irrelevant details, and consolidate multiple related thoughts into higher-level abstractions. A specialized trajectory synthesis pipeline—trained via supervised learning or reinforcement learning—learns to emit optimal memory scheduling decisions by observing which compressions lead to correct downstream reasoning. The model's hidden states during decoding incorporate memory reads, allowing previously compressed information to influence future token generation without storing full thought sequences.
Production Impact
For production LLM systems, this addresses a major cost driver: inference latency and memory consumption for long reasoning tasks. Adopting LightThinker++ would reduce peak memory requirements during long chain-of-thought inference (potentially 30-50% savings based on compression ratios, though the paper abstract doesn't provide exact numbers), directly lowering GPU memory pressure and enabling larger batch sizes or longer sequences on fixed hardware. The trade-off is added complexity: you must integrate the memory primitive scheduler into your inference engine, retrain or fine-tune the LLM to learn when to use these primitives, and maintain logic for memory reads/writes during decoding. For applications like multi-step reasoning, code generation, or planning tasks, this could provide significant latency improvements. However, for simple single-turn queries or streaming applications where you can't afford the overhead of learning memory schedules, the added inference-time cost of managing memory operations may outweigh benefits.
Limitations and When Not to Use This
The paper assumes that intermediate reasoning steps can be meaningfully compressed without losing logical information—an assumption that may fail for reasoning tasks with deeply interdependent steps where seemingly minor details cascade into errors. The method requires training a trajectory synthesis pipeline on reasoning traces, which depends on having high-quality (long-chain) reasoning supervision; this is expensive to obtain and may not generalize across reasoning domains. The paper abstract cuts off mid-sentence and doesn't report quantitative results, so real performance gains, compression ratios, and failure rates remain unclear. Additionally, the approach introduces added latency for memory scheduling decisions during inference—if these scheduling operations are not themselves efficient, the overall speedup may be marginal or negative on shorter reasoning chains.
Research Context
This work builds on the long chain-of-thought prompting literature (Wei et al., 2022) and reasoning efficiency research, extending beyond simple prompt-based approaches to model-level optimization. It advances the efficiency frontier beyond quantization and pruning by addressing the semantic redundancy in reasoning traces themselves. The research connects to memory-augmented neural networks and differentiable memory systems (Neural Turing Machines, Transformers with external memory) by making memory operations learnable and task-specific. This opens research directions toward learned memory hierarchies for LLMs, investigating which domains benefit most from adaptive compression, and extending memory management to multi-step planning and long-context reasoning tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::