Emergent Compositional Communication for Latent World Properties
| Authors | Tomek Kaszyński |
| Year | 2026 |
| HF Upvotes | 7 |
| arXiv | 2604.03266 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Can multi-agent communication pressure extract discrete, compositional representations of invisible physical properties from frozen video features? We show that agents communicating through a Gumbel-Softmax bottleneck with iterated learning develop positionally disentangled protocols for latent properties (elasticity, friction, mass ratio) without property labels or supervision on message structure. With 4 agents, 100% of 80 seeds converge to near-perfect compositionality (PosDis=0.999, holdout 98.3%). Controls confirm multi-agent structure -- not bandwidth or temporal coverage -- drives this effect. Causal intervention shows surgical property disruption (~15% drop on targeted property, <3% on others). A controlled backbone comparison reveals that the perceptual prior determines what is communicable: DINOv2 dominates on spatially-visible ramp physics (98.3% vs 95.1%), while V-JEPA 2 dominates on dynamics-only collision physics (87.4% vs 77.7%, d=2.74). Scale-matched (d=3.37) and frame-matched (d=6.53) controls attribute this gap entirely to video-native pretraining. The frozen protocol supports action-conditioned planning (91.5%) with counterfactual velocity reasoning (r=0.780). Validation on Physics 101 real camera footage confirms 85.6% mass-comparison accuracy on unseen objects, temporal dynamics contributing +11.2% beyond static appearance, agent-scaling compositionality replicating at 90% for 4 agents, and causal intervention extending to real video (d=1.87, p=0.022).
Engineering Breakdown
Plain English
This paper demonstrates that multi-agent systems can learn discrete, compositional communication protocols to describe hidden physical properties (like elasticity, friction, and mass ratio) by watching frozen video features, without any explicit labels or supervision on what they should communicate. The agents communicate through a Gumbel-Softmax bottleneck and use iterated learning, a mechanism inspired by how human language evolves. Results show that with just 4 agents, 100% of 80 experimental runs converged to nearly perfect compositionality (PosDis score of 0.999) with 98.3% accuracy on held-out properties. Ablation studies confirm that multi-agent communication structure—not simply having more bandwidth or temporal coverage—is what drives the emergence of these structured protocols.
Core Technical Contribution
The core novelty is showing that discrete, compositional communication emerges without any supervisory signal on message structure, labels, or what properties should be communicated. Prior work on emergent communication typically required explicit reward signals aligned with human-defined concepts; this paper shows communication pressure alone in a multi-agent setup extracts latent properties from frozen visual features. The use of a Gumbel-Softmax bottleneck forces discrete tokenization, and iterated learning (where communication outputs become inputs for the next agent generation) provides the mechanism for refinement. The causal intervention result—surgically disrupting one property causes ~15% performance drop on that property but <3% on others—demonstrates the agents learn genuinely disentangled representations, not just correlated features.
How It Works
The system takes frozen video features (pre-extracted visual representations) as input to a neural encoder that produces hidden state representations of the scene. Two or more agents are trained simultaneously: one agent (the sender) observes the frozen features and must communicate about the underlying physical properties through a discrete bottleneck (Gumbel-Softmax with temperature annealing), producing a discrete message. A receiver agent sees only this discrete message and must predict the property values from it. Iterated learning means the receiver's learned representation becomes the perceptual input for the next sender in the chain, creating evolutionary pressure toward reusable, efficient communication. The training objective is to minimize prediction error on latent properties while maintaining discrete, structured messages. Over iterations, agents converge to using distinct message tokens for different properties, and combinations of tokens for different values—true compositionality like natural language.
Production Impact
For engineers building multimodal perception systems, this approach offers a way to extract interpretable, disentangled representations of hidden physical properties without manual annotation. In robotic systems, you could learn properties like object elasticity or friction from video alone, enabling better prediction of physical interactions. The compositionality angle is particularly valuable: if your communication protocol is structured (e.g., token-1 encodes material, token-2 encodes size), you get interpretability and potential for transfer—agents trained on one object class might generalize to new objects. However, this comes with significant trade-offs: the method requires multi-agent training loops (higher compute cost than single-model baselines), assumes frozen pre-extracted visual features (so the perceptual prior matters heavily), and has only been validated on synthetic or controlled settings. The 98.3% accuracy is strong but on held-out examples from the same distribution; robustness to real-world sensor noise and out-of-distribution properties remains unproven.
Limitations and When Not to Use This
The paper assumes access to frozen, pre-extracted video features and shows that the perceptual backbone's prior heavily constrains what properties can be discovered—meaning garbage-in features lead to garbage-out communication, with no clear way to detect this failure. The evaluation is limited to 80 seeds of a synthetic setting with just 4 agents and 3 properties; scalability to 10+ agents or 20+ properties is untested. The iterated learning process adds training complexity and computational cost, and it's unclear how many iterations are needed in practice or how sensitive the method is to hyperparameters like temperature schedules. The causal intervention (property disruption) is elegant but still qualitative; the paper lacks comparison to supervised disentanglement baselines (like β-VAE or Factor-VAE) which have well-established metrics. Finally, the paper doesn't address what happens when properties are continuous or on a spectrum, or when multiple properties are correlated—the method may struggle with property entanglement in more realistic domains.
Research Context
This work sits at the intersection of emergent communication (studying how agents develop language-like protocols) and unsupervised representation learning (extracting meaning from raw data without labels). It builds on prior emergent communication research that showed agents can develop signaling systems, but extends it by proving compositional structure can emerge without explicit pressure for compositionality. The paper connects to iterated learning theory from cognitive science and linguistics—the idea that repeated communication-and-reproduction cycles shape structure. It's relevant to recent work on world models and latent property prediction, showing that communication bottlenecks can be as effective as explicit disentanglement objectives. The PosDis (Positional Disentanglement) metric is likely adapted from existing compositionality measures in emergent communication. This opens future directions: combining with active learning so agents query properties dynamically, scaling to continuous properties, and testing generalization to new agents or new object categories.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::