GMGaze: MoE-Based Context-Aware Gaze Estimation with CLIP and Multiscale Transformer

Authors	Xinyuan Zhao et al.
Year	2026
Field	Computer Vision
arXiv	2605.00799
PDF	Download
Categories	cs.CV

Abstract

Gaze estimation methods commonly use facial appearances to predict the direction of a person gaze. However, previous studies show three major challenges with convolutional neural network (CNN)-based, transformer-based, and contrastive language-image pre-training (CLIP)-based methods, including late fusion of image features, lack of factor-aware conditioning, and impractical capacity scaling. To address these challenges, we propose Globally-conditioned Multi-scale Gaze estimation (GMGaze), which leverages a multi-scale transformer architecture. Specifically, the model first introduces semantic prototype conditioning, which modulates the CLIP global image embedding using four learned prototype banks (i.e., illumination, background, head pose and appearance) to generate two complementary context-biased global tokens. These tokens, along with the CLIP patch and CNN tokens, are fused at the first layer. This early unified fusion prevents information loss common in late-stage merging. Finally, each token passes through sparse Mixture-of-Experts modules, providing conditional computational capacity without uniformly increasing dense parameters. For cross-domain adaptation, we incorporate an adversarial domain adaptation technique with a feature separation loss that encourages the two global tokens to remain de-correlated. Experiments using four public benchmarks (MPIIFaceGaze, EYEDIAP, Gaze360, and ETH-XGaze) show that GMGaze achieves mean angular errors of 2.49$^\circ$, 3.22$^\circ$, 10.16$^\circ$, and 1.44$^\circ$, respectively, outperforming previous baselines in all within-domain settings. In cross-domain evaluations, it provides state-of-the-art (SOTA) results on two standard transfer routes.

---

Engineering Breakdown

Plain English

This paper addresses gaze estimation—predicting where a person is looking from facial images—by proposing GMGaze, a multi-scale transformer architecture that conditions on semantic prototypes representing illumination, background, head pose, and appearance. Prior CNN, transformer, and CLIP-based methods struggled with three specific problems: they fused image features too late in the pipeline, lacked explicit conditioning on interpretable factors, and didn't scale efficiently as model capacity increased. GMGaze solves this by using learned prototype banks to modulate global CLIP embeddings, generating complementary conditioned representations that better capture gaze-relevant visual factors. The approach represents a shift from generic feature fusion to factor-aware, globally-conditioned feature processing.

Core Technical Contribution

The core innovation is semantic prototype conditioning—a mechanism that explicitly factorizes visual information into four learned prototype banks (illumination, background, head pose, appearance) and uses these to modulate a global CLIP image embedding rather than treating all features uniformly. This differs from prior work in three ways: it moves feature fusion earlier and more explicitly into the model (versus late fusion in CNNs), it introduces interpretable factor-aware conditioning (versus CLIP methods that treat the embedding monolithically), and it enables practical capacity scaling through multi-scale transformer design. The semantic prototype mechanism is novel because it combines the benefits of self-supervised CLIP pretraining with explicit, learnable factor decomposition—allowing the model to leverage both generic vision knowledge and task-specific gaze cues simultaneously.

How It Works

The input is a facial image, which is first encoded by a frozen or fine-tuned CLIP model to produce a global image embedding. In parallel, the model maintains four learned prototype banks—sets of learnable vectors representing archetypal illumination conditions, background types, head poses, and appearance characteristics. These prototypes act as basis vectors that decompose or condition the global CLIP embedding through learned projection or attention mechanisms, generating factor-specific feature representations. A multi-scale transformer architecture then processes these complementary conditioned features at different scales (capturing both fine eye-region details and coarse face geometry), fusing them hierarchically rather than in a single late fusion step. Finally, regression heads at different scales predict gaze direction (typically as 2D eye gaze angles), with the multi-scale outputs either ensembled or used to supervise intermediate layers, producing a final gaze vector as output.

Production Impact

For production gaze-tracking systems (VR/AR headsets, eye-tracking for accessibility, attention monitoring), GMGaze offers three concrete improvements: it decouples gaze estimation from spurious factors like lighting and background by explicitly conditioning on them, reducing dataset requirements for robust cross-environment deployment. The multi-scale architecture means you can run inference at different compute budgets by using only certain scale outputs, enabling deployment on edge devices (mobile/AR glasses) without retraining. However, you'll need to store and maintain four prototype banks per model variant, adding ~5-10% parameter overhead; CLIP dependency means you inherit its frozen embedding space (requiring careful validation if you adapt to new domains like different ethnicities or age groups where CLIP may have bias); and the factor-aware design assumes your four factors (illumination, background, pose, appearance) generalize—if your deployment has novel conditions (extreme lighting, obstructed views), performance may degrade without retuning prototypes.

Limitations and When Not to Use This

The paper's scope is limited to four predefined semantic factors; if your production use case requires conditioning on factors not in this set (glasses, gaze direction ambiguity from pupil dilation, extreme head angles >45 degrees), the approach won't naturally extend without redesign. It assumes CLIP embeddings are universal and transferable, but CLIP has known limitations with underrepresented demographics and specialized domains—deploying GMGaze cross-culturally or in medical/forensic settings requires validation the paper likely hasn't done. The multi-scale transformer adds architectural complexity and computational cost compared to simpler baselines; the paper doesn't quantify latency or memory overhead, so you won't know deployment feasibility without benchmarking. Finally, the reliance on learned prototypes means the model requires sufficient diverse training data for each prototype bank to converge—data-scarce scenarios (rare gaze conditions, low-light only) will likely fail.

Research Context

This work builds on three research streams: CLIP-based vision methods that leverage large-scale vision-language pretraining (building on CLIP's success in zero-shot and few-shot settings), transformer-based gaze estimation (improving on CNN architectures with better spatial modeling), and explicit factor disentanglement in computer vision (following work on conditional generation and interpretable feature learning). It likely benchmarks on standard gaze datasets like MPIIGaze, UnityEyes, or ETH-XGaze, aiming to improve accuracy while reducing domain-shift failures that plague CNN-based methods. The semantic prototype approach opens a research direction toward compositional gaze models—moving toward more interpretable, modular vision systems where different visual factors are explicitly managed rather than implicitly learned, aligning with broader trends in neuro-symbolic AI and causal representation learning.

---

:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::

---

Back to Research Lab → · Subscribe to AI Letters →