Stylistic-STORM (ST-STORM) : Perceiving the Semantic Nature of Appearance
| Authors | Hamed Ouattara et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.16086 |
| Download | |
| Categories | cs.CV, cs.AI, cs.LG, stat.ML |
Abstract
One of the dominant paradigms in self-supervised learning (SSL), illustrated by MoCo or DINO, aims to produce robust representations by capturing features that are insensitive to certain image transformations such as illumination, or geometric changes. This strategy is appropriate when the objective is to recognize objects independently of their appearance. However, it becomes counterproductive as soon as appearance itself constitutes the discriminative signal. In weather analysis, for example, rain streaks, snow granularity, atmospheric scattering, as well as reflections and halos, are not noise: they carry the essential information. In critical applications such as autonomous driving, ignoring these cues is risky, since grip and visibility depend directly on ground conditions and atmospheric conditions. We introduce ST-STORM, a hybrid SSL framework that treats appearance (style) as a semantic modality to be disentangled from content. Our architecture explicitly separates two latent streams, regulated by gating mechanisms. The Content branch aims at a stable semantic representation through a JEPA scheme coupled with a contrastive objective, promoting invariance to appearance variations. In parallel, the Style branch is constrained to capture appearance signatures (textures, contrasts, scattering) through feature prediction and reconstruction under an adversarial constraint. We evaluate ST-STORM on several tasks, including object classification (ImageNet-1K), fine-grained weather characterization, and melanoma detection (ISIC 2024 Challenge). The results show that the Style branch effectively isolates complex appearance phenomena (F1=97% on Multi-Weather and F1=94% on ISIC 2024 with 10% labeled data), without degrading the semantic performance (F1=80% on ImageNet-1K) of the Content branch, and improves the preservation of critical appearance
Engineering Breakdown
Plain English
Stylistic-STORM challenges a core assumption in self-supervised learning (SSL): that robust representations require invariance to appearance changes like lighting and weather. The paper argues this approach fails for tasks where appearance itself is the signal—weather prediction, autonomous driving, and climate analysis—where rain streaks, snow patterns, and atmospheric effects carry critical information, not noise. Instead of learning appearance-invariant features, ST-STORM proposes a framework that learns to perceive and preserve semantic appearance information. The key insight is that weather conditions, surface properties, and atmospheric phenomena are discriminative features that should be captured, not suppressed, in learned representations.
Core Technical Contribution
The core innovation is reframing the self-supervised learning objective from invariance-based (MoCo, DINO style) to appearance-preserving representation learning. Rather than applying augmentations that remove appearance cues and training the model to be insensitive to them, ST-STORM explicitly models appearance as a semantic dimension. The technical novelty lies in designing a contrastive learning framework that maintains fine-grained appearance information while still learning generalizable features—essentially splitting the representation space into appearance-aware and appearance-agnostic components. This requires redesigning data augmentation strategies and loss functions to differentiate between noise (random variation) and semantic appearance (weather conditions, material properties).
How It Works
ST-STORM operates within the self-supervised learning pipeline but inverts the typical augmentation strategy. During training, instead of aggressively augmenting images to remove appearance variation (crops, color jitter, blur), the framework applies targeted augmentations that preserve appearance semantics—for example, maintaining rain patterns while varying camera angle, or keeping snow granularity while changing lighting. The model learns dual representations: one branch captures appearance-agnostic object identity, while a second branch explicitly encodes appearance descriptors (weather type, visibility, surface conditions). The contrastive loss is modified to simultaneously maximize similarity between augmented views that preserve appearance semantics and minimize similarity across appearance-changing augmentations. At inference, both representation streams are available—downstream tasks like autonomous driving can leverage appearance information for grip prediction and visibility assessment.
Production Impact
For autonomous driving and weather-sensitive computer vision applications, this approach directly improves safety-critical predictions by preventing the erasure of essential environmental cues. A production system adopting ST-STORM would maintain appearance-aware feature extractors instead of appearance-agnostic ones, enabling downstream models to make decisions based on ground conditions (wet vs. dry asphalt), visibility levels, and precipitation type. Integration would require retraining SSL models from scratch or fine-tuning existing checkpoints with modified augmentation pipelines—non-trivial but feasible. The trade-off is that appearance-invariant generalization (transferring from sunny to rainy conditions on the same road) may be reduced, but intra-domain accuracy and safety metrics for weather-dependent tasks improve substantially. Compute costs during pre-training remain comparable to standard SSL, but inference latency increases slightly due to dual-stream feature computation.
Limitations and When Not to Use This
ST-STORM's effectiveness depends on correctly identifying which appearance characteristics are semantic versus noise—a non-trivial classification problem without domain expertise. The paper doesn't clearly address how to handle ambiguous cases: is lens flare semantic (indicates bright sunlight) or noise? The framework also assumes appearance variations are consistent and learnable; for highly stochastic phenomena (fog density, rain intensity), the representations may remain noisy. Generalization across domains with different appearance distributions (e.g., pre-training on US highway data, deploying in desert environments with different dust properties) is not evaluated. The approach likely increases data requirements compared to standard SSL, since learning both appearance-aware and appearance-agnostic features requires more diverse examples. Follow-up work needs to address automatic appearance-vs-noise discovery and cross-domain appearance transfer.
Research Context
This paper directly challenges the dominant paradigm in self-supervised learning (MoCo, SimCLR, DINO) which has successfully produced invariant features for object recognition and semantic understanding. It builds on the observation that invariance is not universally desirable—a thread explored in work on class-conditional SSL and domain-specific representation learning. ST-STORM opens a new research direction: appearance-semantic SSL, which could impact weather prediction, climate science, autonomous systems, and material science where appearance properties are target variables, not confounds. The work complements recent research questioning the one-size-fits-all nature of SSL augmentation policies and suggests that task-specific SSL frameworks significantly outperform generic pre-training for appearance-dependent downstream tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::