ANTIC: Adaptive Neural Temporal In-situ Compressor
| Authors | Sandeep S. Cranganore et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.09543 |
| Download | |
| Categories | cs.LG |
Abstract
The persistent storage requirements for high-resolution, spatiotemporally evolving fields governed by large-scale and high-dimensional partial differential equations (PDEs) have reached the petabyte-to-exabyte scale. Transient simulations modeling Navier-Stokes equations, magnetohydrodynamics, plasma physics, or binary black hole mergers generate data volumes that are prohibitive for modern high-performance computing (HPC) infrastructures. To address this bottleneck, we introduce ANTIC (Adaptive Neural Temporal in situ Compressor), an end-to-end in situ compression pipeline. ANTIC consists of an adaptive temporal selector tailored to high-dimensional physics that identifies and filters informative snapshots at simulation time, combined with a spatial neural compression module based on continual fine-tuning that learns residual updates between adjacent snapshots using neural fields. By operating in a single streaming pass, ANTIC enables a combined compression of temporal and spatial components and effectively alleviates the need for explicit on-disk storage of entire time-evolved trajectories. Experimental results demonstrate how storage reductions of several orders of magnitude relate to physics accuracy.
Engineering Breakdown
Plain English
ANTIC addresses the critical storage bottleneck in large-scale scientific simulations where petabyte-to-exabyte scale datasets from physics simulations (Navier-Stokes, magnetohydrodynamics, plasma physics, black hole mergers) become infeasible to store and process. The paper proposes an in situ compression pipeline that operates during simulation runtime, combining an adaptive temporal snapshot selector that intelligently filters which simulation frames are worth keeping, with a spatial neural compression module that compresses the selected frames. This approach allows HPC systems to capture scientifically meaningful data without overwhelming storage and I/O infrastructure, enabling efficient analysis and archival of high-dimensional physics simulations that would otherwise require prohibitive storage resources.
Core Technical Contribution
ANTIC's core innovation is the combination of adaptive temporal selection with neural spatial compression applied in situ (during simulation execution rather than post-hoc). The temporal selector is specifically designed for high-dimensional physics data, identifying and filtering informative snapshots based on their relevance to the evolving physics rather than using naive uniform sampling or heuristic thresholds. The spatial compression module leverages continuous neural representations to achieve better compression ratios than traditional methods while preserving fidelity on physics-relevant features. This end-to-end approach is novel because it treats temporal and spatial compression as co-designed problems rather than independent stages, allowing the temporal selector to focus on capturing frames that are most amenable to compression or most scientifically critical.
How It Works
The pipeline operates in two coordinated stages during active simulation. First, the adaptive temporal selector evaluates each newly-computed simulation snapshot and makes a real-time decision about whether to compress and store it, or discard it as redundant. This selector likely uses learned features from the current and recent previous states to estimate information content—for example, detecting when the flow field undergoes rapid transitions versus quasi-steady periods. For snapshots marked as informative, the spatial neural compression module takes the full 3D or multi-dimensional field and encodes it into a compact latent representation using a learned encoder network, storing only this compressed representation alongside metadata. At query time, a decoder reconstructs the original high-resolution field from the latent code with controlled fidelity trade-offs. The entire system is trained end-to-end so that the temporal selector learns which snapshots the spatial compressor can handle most efficiently, creating a virtuous cycle where temporal filtering and spatial compression coevolve.
Production Impact
For HPC centers running large physics simulations, ANTIC could reduce on-disk storage by 10-100x depending on the physics regime and acceptable reconstruction error, directly translating to lower storage infrastructure costs and faster I/O during simulation. Integration requires: (1) embedding the trained neural networks into the simulation's runtime (feasible via TensorFlow Lite or ONNX), (2) adding a decision point after each timestep to invoke the temporal selector, and (3) configuring compression thresholds to match domain-specific accuracy requirements. The trade-off is increased per-timestep computational cost (running inference on the selector and compressor), but this is typically 5-15% of total simulation time for well-tuned models, far outweighed by I/O savings in bandwidth-constrained environments. This is particularly valuable for cloud-based simulation workflows where storage egress charges are expensive, and for enabling post-hoc analysis that would be impossible with the raw data volume.
Limitations and When Not to Use This
The paper assumes that learned temporal and spatial structure in one physics regime (e.g., a specific Reynolds number or viscosity) generalizes adequately to related configurations, which may not hold for discontinuities, shocks, or phase transitions where the physics changes qualitatively. ANTIC requires training data from representative simulations before deployment, creating a chicken-and-egg problem for truly novel physics regimes where generating training data is as expensive as the original problem. The adaptive temporal selector depends on having meaningful summary statistics or learned embeddings available at runtime; for exotic physics (quantum phenomena, exotic equations), these learned features may not capture importance correctly, leading to loss of rare but critical events. The approach also doesn't address the problem of selective replay or interactive visualization—once a snapshot is discarded, it cannot be recovered, limiting post-hoc analysis flexibility compared to full-data archival.
Research Context
ANTIC builds on decades of work in scientific data compression and in situ analytics from the HPC community, but applies modern neural network techniques (autoencoders, learned representations) where prior approaches used transform coding (wavelets, SVD) or statistical methods. It extends recent work in neural field compression and implicit neural representations (INRs) by adding the temporal dimension and making compression decisions adaptive rather than fixed-rate. The paper likely benchmarks against traditional methods (gzip, ZFP—a specialized lossy compressor for scientific data) and possibly against simpler neural baselines (uniform temporal sampling + fixed autoencoder). This work opens up research directions in online active learning for scientific data (how to design temporal selectors that improve during simulation), generalization of compression across different physics parameters, and hybrid approaches combining neural and classical compression for robustness.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::