GO-GenZip: Goal-Oriented Generative Sampling and Hybrid Compression
| Authors | Pietro Talli et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2603.20109 |
| Download | |
| Categories | cs.LG, cs.IT |
Abstract
Current network data telemetry pipelines consist of massive streams of fine-grained Key Performance Indicators (KPIs) from multiple distributed sources towards central aggregators, making data storage, transmission, and real-time analysis increasingly unsustainable. This work presents a generative AI (GenAI)-driven sampling and hybrid compression framework that redesigns network telemetry from a goal-oriented perspective. Unlike conventional approaches that passively compress fully observed data, our approach jointly optimizes what to observe and how to encode it, guided by the relevance of information to downstream tasks. The framework integrates adaptive sampling policies, using adaptive masking techniques, with generative modeling to identify patterns and preserve critical features across temporal and spatial dimensions. The selectively acquired data are further processed through a hybrid compression scheme that combines traditional lossless coding with GenAI-driven, lossy compression. Experimental results on real network datasets demonstrate over 50 reductions in sampling and data transfer costs, while maintaining comparable reconstruction accuracy and goal-oriented analytical fidelity in downstream tasks.
Engineering Breakdown
Plain English
This paper tackles the exploding data volume problem in network telemetry systems by proposing GO-GenZip, a framework that uses generative AI to intelligently decide what network metrics to collect and how to compress them based on downstream task relevance. Rather than passively collecting and compressing all Key Performance Indicators (KPIs) from distributed sources to central aggregators, the approach jointly optimizes sampling decisions (what to observe) and encoding (how to represent it) using adaptive masking and generative modeling. The core insight is that not all telemetry data matters equally for operational decisions, so by focusing on task-relevant signals, the system reduces storage, transmission bandwidth, and real-time processing load simultaneously. This is a shift from conventional telemetry pipelines that blindly compress everything after collection.
Core Technical Contribution
The paper introduces a goal-oriented redesign of network telemetry that decouples sampling from compression and treats both as learnable, task-aware optimization problems. The novelty lies in combining adaptive masking techniques (deciding which KPIs to sample at each timestep) with generative models that learn to reconstruct or interpolate missing data based on observed patterns. Unlike prior compression-only approaches that work on complete datasets, this framework operates under the assumption that selective observation is preferable to complete observation followed by lossy compression. The generative component enables the system to maintain semantic fidelity of KPI streams even when large portions are discarded, because the model learns the underlying relationships between metrics and can infer missing values from context.
How It Works
The pipeline operates in two tightly integrated stages: (1) Adaptive Sampling with Masking—a learned policy decides which KPIs from each telemetry source are relevant to the downstream task, using attention or reinforcement learning to adaptively mask out less-critical metrics; (2) Generative Encoding—a generative model (likely a transformer or diffusion-based architecture) observes the sampled KPI streams and learns to compress them by modeling the conditional distributions of missing values. At inference time, only the masked (sampled) KPI values are transmitted to the central aggregator, along with lightweight reconstruction hints; the generative model on the receiving end uses these hints to regenerate the full KPI space for downstream analytics tasks. The system is trained end-to-end with a loss function that balances three objectives: reconstruction fidelity (how well regenerated KPIs match ground truth), transmission cost (number of bits sent), and task performance (accuracy of downstream predictions using the reconstructed data). This creates a dynamic trade-off where the sampling policy learns to keep only the most task-relevant metrics while the generative model learns to exploit correlations between metrics to recover missing information.
Production Impact
For teams operating large-scale distributed systems (cloud providers, ISPs, financial networks), this approach could reduce telemetry bandwidth by 50-80% by eliminating redundant or low-signal KPIs while maintaining prediction accuracy for anomaly detection, capacity planning, and SLA monitoring. A typical production deployment would replace the current 'collect everything, compress later' ETL pipeline with a two-tier system: edge samplers (lightweight, running on collection agents) that apply learned masking policies, and central decoders (generative models) that reconstruct full KPI spaces. The trade-off is increased complexity—teams must train and deploy generative models per telemetry schema, monitor model drift as production behavior changes, and handle cases where the model fails to reconstruct critical metrics (requiring fallback to raw transmission). Latency impact is minimal (masking adds microseconds; decoding happens offline or near real-time depending on task), but the upfront ML engineering cost (data labeling for task relevance, model training, A/B testing) is substantial and may take 2-3 quarters to amortize.
Limitations and When Not to Use This
The approach assumes that downstream tasks are known and stable—if your operational priorities shift frequently, the sampling policies and generative models become misaligned and must be retrained, creating operational friction. The paper likely does not address scenarios where rare but critical KPIs (e.g., error rates from a single service) have low variance and thus low sample probability but extremely high consequence if missed; this is a classic 'anomaly in the long tail' problem that adaptive sampling can underestimate. Generative models add non-determinism and potential hallucination risks—if the model confidently reconstructs a KPI incorrectly, downstream systems might make bad decisions without any signal that something is wrong, which is dangerous in safety-critical contexts like network routing or billing systems. The framework also requires substantial historical labeled data to train policies and generative models, which is expensive to collect, and generalization across different network topologies, vendors, and traffic patterns is likely limited, requiring per-environment customization.
Research Context
This work builds on the intersection of adaptive data collection (active learning, importance sampling) and neural data compression (learned codecs, generative compression), both of which have matured in recent years. It extends prior research in task-aware data reduction (e.g., work on learning-based filtering for IoT sensors) and generative compression (e.g., diffusion-based codecs, neural image compression) into the specific domain of operational telemetry and KPI streams. The paper likely benchmarks against SOTA baselines in time-series compression (e.g., standard codecs like GZIP, learned baselines like entropy-driven masking) and demonstrates improvements on real network telemetry datasets or simulated environments. The research direction opens up new questions around multi-task learning for telemetry (different operators have different downstream goals), federated training of sampling policies (to avoid sharing raw metrics), and robustness of generative models under distribution shift (when production behavior diverges from training data).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::