Event-Driven Temporal Graph Networks for Asynchronous Multi-Agent Cyber Defense in NetForge_RL
| Authors | Igor Jankowski |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.09523 |
| Download | |
| Categories | cs.LG, cs.MA |
Abstract
The transition of Multi-Agent Reinforcement Learning (MARL) policies from simulated cyber wargames to operational Security Operations Centers (SOCs) is fundamentally bottlenecked by the Sim2Real gap. Legacy simulators abstract away network protocol physics, rely on synchronous ticks, and provide clean state vectors rather than authentic, noisy telemetry. To resolve these limitations, we introduce NetForge_RL: a high-fidelity cyber operations simulator that reformulates network defense as an asynchronous, continuous-time Partially Observable Semi-Markov Decision Process (POSMDP). NetForge enforces Zero-Trust Network Access (ZTNA) constraints and requires defenders to process NLP-encoded SIEM telemetry. Crucially, NetForge bridges the Sim2Real gap natively via a dual-mode engine, allowing high-throughput MARL training in a mock hypervisor and zero-shot evaluation against live exploits in a Docker hypervisor. To navigate this continuous-time POSMDP, we propose Continuous-Time Graph MARL (CT-GMARL), utilizing fixed-step Neural Ordinary Differential Equations (ODEs) to process irregularly sampled alerts. We evaluate our framework against discrete baselines (R-MAPPO, QMIX). Empirical results demonstrate that CT-GMARL achieves a converged median Blue reward of 57,135 - a 2.0x improvement over R-MAPPO and 2.1x over QMIX. Critically, CT-GMARL restores 12x more compromised services than the strongest baseline by avoiding the "scorched earth" failure mode of trivially minimizing risk by destroying network utility. On zero-shot transfer to the live Docker environment, CT-GMARL policies achieve a median reward of 98,026, validating the Sim2Real bridge.
Engineering Breakdown
Plain English
This paper addresses a critical gap in deploying multi-agent reinforcement learning (MARL) systems from simulation into real Security Operations Centers (SOCs). The authors introduce NetForge_RL, a high-fidelity cyber operations simulator that models network defense as an asynchronous, continuous-time Partially Observable Semi-Markov Decision Process (POSMDP) rather than the synchronous, discrete-tick abstractions used in legacy simulators. NetForge incorporates realistic constraints like Zero-Trust Network Access (ZTNA) policies and processes natural language-encoded SIEM telemetry instead of clean state vectors, bringing simulation substantially closer to real operational conditions. The dual-mode engine architecture allows policies trained in high-fidelity simulation to transfer more effectively to production SOCs, directly addressing the Sim2Real gap that has prevented MARL adoption in critical security operations.
Core Technical Contribution
The core innovation is reformulating cyber defense from a standard Markov Decision Process (MDP) into a Partially Observable Semi-Markov Decision Process (POSMDP) within a high-fidelity, asynchronous simulator. This is significant because it captures two aspects of real SOCs that prior simulators ignored: (1) defenders operate under partial observability with noisy, incomplete telemetry rather than perfect state knowledge, and (2) time is continuous and event-driven rather than progressing in discrete synchronized ticks. The dual-mode engine is the architectural breakthrough—it allows seamless switching between simulation and real network environments, enabling iterative fine-tuning of learned policies without the catastrophic policy degradation typically seen when transferring from synthetic to real data. This natively addresses Sim2Real gap through architecture rather than post-hoc domain adaptation techniques.
How It Works
NetForge operates as a simulator that accepts network topology, defender actions, and adversarial attack sequences, then produces observations formatted as NLP-encoded SIEM alerts rather than raw numerical state vectors. The POSMDP formulation means the state space is only partially observable to the agent—the defender receives noisy, delayed alerts reflecting real network telemetry rather than ground truth information about attacker positions or asset compromise status. The asynchronous, continuous-time engine processes events as they occur (network traffic, security alerts, attacker movements) without waiting for synchronized clock ticks, which more faithfully models real SOC operations where events arrive unpredictably. Actions taken by the MARL agent must satisfy Zero-Trust Network Access (ZTNA) constraints, meaning every access request requires explicit authorization and cannot assume trust based on network location or prior authentication. The dual-mode engine allows the same trained policy to execute against both the high-fidelity simulation and real network infrastructure, with a thin adapter layer translating between simulation outputs and actual security tools (firewalls, intrusion detection systems, access controls). During training, gradient updates and policy optimization happen within the simulation environment, but the learned weights transfer directly to production with minimal retraining.
Production Impact
For engineers deploying MARL in SOCs, NetForge eliminates the multi-month gap between lab results and operational deployment—policies can be trained in high-fidelity simulation and deployed with confidence because the simulation faithfully reproduces real network conditions (asynchrony, partial observability, noisy telemetry). This reduces the risk of catastrophic policy failures that have plagued prior sim-to-real transitions in security, where a policy optimized for clean synthetic data performs poorly against real alerts. The requirement to process NLP-encoded SIEM telemetry means your training pipeline must include robust NLP components and alert parsing, adding complexity but substantially improving transferability since real SOCs already generate alerts in natural language. Integration cost is moderate: you need to build adapters between your simulator and existing SIEM/network tools, implement the POSMDP environment, and instrument your training loop to enforce ZTNA constraints, but the architecture is designed for this integration. The trade-off is computational overhead—asynchronous, continuous-time simulation is more expensive than discrete-tick simulation, so you'll need more GPU/CPU resources during training, but this is offset by dramatically reduced operational risk and fewer expensive real-world test-and-debug cycles.
Limitations and When Not to Use This
NetForge assumes that NLP-encoded SIEM alerts are sufficient to represent the true security state, but sophisticated attacks may bypass alerting entirely or generate misleading alerts, leaving the defender with a fundamentally incomplete picture regardless of simulation fidelity. The paper doesn't address adversarial adaptation—real attackers respond to defensive policies over time, but the simulator likely uses fixed or scripted adversary models that don't co-evolve with defender strategies, potentially leading to policies that fail when faced with truly adaptive attackers. The ZTNA constraint enforcement is only as good as the constraint specification itself; if your ZTNA policies are misconfigured or incomplete, the simulator will propagate those errors into the learned policy. The dual-mode engine requires a stable interface between simulation and real networks, which may be fragile in practice due to differences in latency, alert ordering, and network state representation—the paper doesn't deeply explore failure modes when this interface breaks or drifts over time. Additionally, the approach requires substantial labeled data for training the NLP components and SIEM alert parsers, and generalization to organizations with different alert formats or security tool ecosystems is not addressed.
Research Context
This work builds on decades of Sim2Real research in robotics and reinforcement learning, adapting techniques like domain randomization and physics-accurate simulation to the cybersecurity domain where the 'physics' is network protocols and alert generation. It extends prior MARL work (e.g., independent learners, centralized training with decentralized execution) into the security space while addressing the unique challenge of partial observability through the POSMDP formulation, which has theoretical roots in the POMDP literature but is less commonly applied to multi-agent security scenarios. The emphasis on asynchronous, continuous-time models challenges the discrete-timestep assumption that has dominated RL benchmarks and opens a research direction toward event-driven RL environments. NetForge positions itself as a benchmark and environment for cyber defense research—similar to how OpenAI Gym or ALE have catalyzed RL research in other domains, this simulator could standardize how the research community evaluates MARL policies for cybersecurity, enabling reproducible comparisons and accelerating the transition from academic prototypes to operational security systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::