Hyper Input Convex Neural Networks for Shape Constrained Learning and Optimal Transport
| Authors | Shayan Hundrieser et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.26942 |
| Download | |
| Categories | cs.LG, stat.ME, stat.ML |
Abstract
We introduce Hyper Input Convex Neural Networks (HyCNNs), a novel neural network architecture designed for learning convex functions. HyCNNs combine the principles of Maxout networks with input convex neural networks (ICNNs) to create a neural network that is always convex in the input, theoretically capable of leveraging depth, and performs reliable when trained at scale compared to ICNNs. Concretely, we prove that HyCNNs require exponentially fewer parameters than ICNNs to approximate quadratic functions up to a given precision. Throughout a series of synthetic experiments, we demonstrate that HyCNNs outperform existing ICNNs and MLPs in terms of predictive performance for convex regression and interpolation tasks. We further apply HyCNNs to learn high-dimensional optimal transport maps for synthetic examples and for single-cell RNA sequencing data, where they oftentimes outperform ICNN-based neural optimal transport methods and other baselines across a wide range of settings.
Engineering Breakdown
Plain English
This paper introduces Hyper Input Convex Neural Networks (HyCNNs), a new neural architecture designed to learn convex functions more efficiently than existing methods. The key innovation combines Maxout networks with Input Convex Neural Networks (ICNNs) to create a model that maintains convexity in its input while theoretically benefiting from network depth. The authors prove HyCNNs need exponentially fewer parameters than ICNNs to approximate quadratic functions to a given precision, and demonstrate superior performance on convex regression, interpolation tasks, and high-dimensional optimal transport map learning compared to standard ICNNs and MLPs.
Core Technical Contribution
The core novelty is the architectural combination of Maxout networks with ICNN principles to create a provably convex-in-input network that scales better than prior approaches. Unlike standard ICNNs which struggle with depth and require many parameters, HyCNNs leverage Maxout's ability to approximate convex functions through positively-weighted combinations of linear functions, maintaining input convexity throughout the network. The theoretical contribution is a proof that HyCNNs achieve exponential parameter efficiency over ICNNs for quadratic function approximation, addressing a fundamental limitation of ICNNs that has hindered their practical adoption at scale.
How It Works
HyCNNs operate by stacking Maxout layers (which compute the maximum across groups of linear transformations) in a way that preserves input convexity throughout the network depth. Each layer applies a set of linear transformations to the input or previous layer's output, then takes the element-wise maximum—this operation is inherently convex when applied to a convex function. The architecture maintains the crucial invariant that the overall function f(x) remains convex in the input x, which is verified through careful design of weight constraints (ensuring non-negative weights where needed). Unlike ICNNs which must restrict their hidden-to-hidden connections to prevent non-convexity, HyCNNs can use full connectivity while preserving the convexity property, enabling both deeper networks and better parameter efficiency.
Production Impact
For engineers building optimization-heavy systems, HyCNNs enable significant parameter reduction when learning convex objectives—critical for embedded systems or resource-constrained environments where ICNNs were previously prohibitive. In applications like optimal transport computation, portfolio optimization, or physical simulations where convexity is essential, HyCNNs provide a principled learned alternative to hand-coded solvers with orders of magnitude fewer parameters. The training procedure appears standard (the paper mentions 'trained at scale'), meaning existing deep learning infrastructure and optimization techniques (Adam, SGD) likely work without modification. The main production trade-off is that this architecture is specialized to convex function learning—it cannot be applied to non-convex problems like standard classification, limiting its applicability to domains where convexity is a requirement or known property of the target function.
Limitations and When Not to Use This
This architecture is fundamentally restricted to learning convex functions; any problem where the target function is non-convex cannot use HyCNNs, eliminating its use in the vast majority of modern deep learning applications (classification, generation, etc.). The paper's evaluation appears to focus on synthetic convex functions and optimal transport, which is a relatively narrow domain—real-world convex optimization problems often have additional structure, noise, or constraints that aren't addressed in the experiments shown. The paper doesn't provide detailed computational complexity analysis comparing wall-clock training time or inference latency versus ICNNs, only theoretical parameter counts, so practical speed improvements on modern hardware remain unclear. Outstanding questions include how HyCNNs handle approximate convexity (when the true function is nearly but not exactly convex), how they scale to very high-dimensional problems (>1000 dimensions), and whether the theoretical guarantees hold under stochastic gradient descent with noisy or finite samples.
Research Context
This work builds directly on Input Convex Neural Networks (ICNNs), which enforced convexity through architectural constraints but suffered from poor parameter efficiency and limited depth. The connection to Maxout networks—which represent convex functions as max-pooling over linear functions—provides the theoretical insight that enables HyCNNs' superior efficiency. The paper positions itself in the intersection of convex optimization and deep learning, addressing a practical gap where problems provably have convex structure but ICNN solutions were too expensive. This opens a research direction for specialized neural architectures tailored to mathematical structure in the problem domain, potentially extending to other function classes with known properties (monotone functions, submodular functions, etc.).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::