GeoStack: A Framework for Quasi-Abelian Knowledge Composition in VLMs
:::info Stub — Full Engineering Breakdown Coming This paper was featured on Hugging Face Daily Papers on 2026-05-07 with 2 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Pranav Mantini & Shishir K. Shah |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2605.06477 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
We address the challenge of knowledge composition in Vision-Language Models (VLMs), where accumulating expertise across multiple domains or tasks typically leads to catastrophic forgetting. We introduce GeoStack (Geometric Stacking), a modular framework that allows independently trained domain experts to be composed into a unified model. By imposing geometric and structural constraints on the adapter manifold, GeoStack ensures the foundational knowledge of the base model is preserved. Furthermore, we mathematically demonstrate a weight-folding property that achieves constant-time inference complexity (O(1)), regardless of the number of integrated experts. Experimental results across multi-domain adaptation and class-incremental learning show that GeoStack provides an efficient mechanism for long-term knowledge composition while significantly mitigating catastrophic forgetting. Code is available at https://github.com/QuantitativeImagingLaboratory/GeoStack.
Engineering Breakdown
Plain English
GeoStack solves a fundamental problem in Vision-Language Models: when you fine-tune a base model on multiple domains or tasks sequentially, it forgets what it learned before (catastrophic forgetting). This paper introduces a modular framework where you train independent domain experts separately, then stack them together while preserving the base model's original knowledge through geometric constraints. The key engineering win is that inference stays O(1) constant-time regardless of how many experts you add, thanks to a weight-folding property.
Key Engineering Insight
The core innovation is treating adapter composition as a geometric problem with structural constraints on the manifold, rather than naive parameter concatenation. This constraint-based approach lets you achieve weight folding—essentially collapsing multiple expert adapters into a single inference pass—which eliminates the typical linear cost scaling you'd expect from stacking multiple models.
Why It Matters for Engineers
In production, you often need to extend VLMs with new domains without retraining from scratch or losing existing capabilities. GeoStack lets you deploy new domain experts incrementally without performance degradation or inference latency penalties. This directly addresses the multi-tenant problem: teams building multi-domain AI systems can now add new capabilities (e.g., medical imaging, satellite analysis) without the usual tradeoff between coverage and speed.
Research Context
Prior work on multi-task VLMs either suffered from catastrophic forgetting or used architectural hacks that scaled inference linearly with expert count. GeoStack advances beyond adapter-based methods by introducing geometric constraints that enforce a quasi-abelian structure on the composition space, enabling mathematical guarantees about knowledge preservation and constant-time inference. This bridges modular composition (popular in recent VLM work) with the mathematical rigor needed for production reliability.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::