Predictive Coding Graphs are a Superset of Feedforward Neural Networks
| Authors | Björn van Zwol |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2603.06142 |
| Download | |
| Categories | cs.LG, cs.AI, cs.NE, stat.ML |
Abstract
Predictive coding graphs (PCGs) are a recently introduced generalization to predictive coding networks, a neuroscience-inspired probabilistic latent variable model. Here, we prove how PCGs define a mathematical superset of feedforward artificial neural networks (multilayer perceptrons). This positions PCNs more strongly within contemporary machine learning (ML), and reinforces earlier proposals to study the use of non-hierarchical neural networks for ML tasks, and more generally the notion of topology in neural networks.
Engineering Breakdown
Plain English
This paper proves that predictive coding graphs (PCGs)—a neuroscience-inspired probabilistic model—mathematically encompass feedforward neural networks (MLPs) as a special case. The authors establish a formal superset relationship, meaning any feedforward network can be expressed within the PCG framework, but PCGs can also represent architectures that standard feedforward networks cannot. This theoretical result bridges neuroscience-inspired probabilistic models with contemporary deep learning, suggesting that network topology and non-hierarchical structures deserve more investigation in practical ML systems.
Core Technical Contribution
The core contribution is a mathematical proof demonstrating that feedforward artificial neural networks are a strict subset of predictive coding graphs. Rather than PCGs being an alternative or competing framework, this work formally establishes that PCGs are strictly more expressive and general. This shifts the narrative from PCGs as a neuroscience curiosity to PCGs as a principled superset that unifies both probabilistic latent variable models and deterministic neural networks under one mathematical umbrella.
How It Works
Predictive coding graphs operate as probabilistic graphical models where nodes represent variables and edges represent dependencies, with a focus on minimizing prediction error through message passing. In contrast to feedforward networks that apply fixed compositions of linear and nonlinear transformations in a single forward pass, PCGs iteratively refine estimates of latent variables by comparing predictions to observations and propagating error signals backward through the graph structure. The key insight is that a feedforward network's deterministic computation—matrix multiplication followed by activation functions—can be reformulated as a special case of PCG inference where the graph has specific constraints: acyclic layered structure, no feedback loops, and point estimates rather than distributions. By removing these constraints (allowing cycles, arbitrary connectivity, and probabilistic inference), PCGs enable richer model classes, including recurrent and non-hierarchical topologies that feedforward networks cannot naturally represent. The paper likely constructs an explicit mapping showing how standard MLP weights and biases translate into PCG graph structure and message-passing rules.
Production Impact
For production systems, this theoretical result primarily provides architectural motivation rather than immediate practical improvements. If adopted, engineers could potentially use PCG frameworks to explore non-hierarchical or recurrent structures more naturally without force-fitting them into feedforward paradigms, potentially discovering more efficient topologies for specific tasks. However, PCG inference typically requires iterative message passing (similar to belief propagation), which is computationally more expensive than a single forward pass, introducing latency overhead—critical for real-time inference in recommendation systems or API endpoints. The superset property means you could gradually migrate existing feedforward models into a PCG framework for experimentation, but you'd need to justify the additional computational cost against any gains in model expressivity or sample efficiency. This is most valuable for research teams exploring novel architectures or domains where standard feedforward networks have known limitations (e.g., structured prediction, reasoning tasks).
Limitations and When Not to Use This
The paper is primarily theoretical and does not demonstrate empirical advantages of PCGs over feedforward networks on standard benchmarks, leaving open whether the additional expressivity translates to practical improvements. The iterative inference required by PCGs makes them slower to deploy than feedforward networks, and the paper likely doesn't address how to set convergence criteria or handle approximate inference in practice. The proof establishes that PCGs are a superset, but this doesn't mean all PCG instances are useful or learnable—a much richer model class can paradoxically become harder to train due to increased expressivity and parameter count. The work assumes you can fully specify the PCG topology and prior structure, which requires domain knowledge; in practice, automatically discovering optimal non-hierarchical topologies remains an open problem.
Research Context
This paper builds on a growing thread in neuroscience-inspired ML that views predictive coding—the brain's hypothesized mechanism for inference—as a unified framework for learning and inference. It relates to prior work on hierarchical predictive coding networks and extends them to arbitrary graph topologies, addressing long-standing criticism that these models were too restrictive. The result reinforces interest in non-hierarchical and recurrent architectures, echoing other research directions like neural architecture search (NAS) and graph neural networks that question the assumption that strictly feedforward layered design is optimal. By formally positioning PCGs within classical ML theory, the paper opens a research direction for exploring when and why non-hierarchical topologies outperform standard feedforward networks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::