Generative Refinement Networks for Visual Synthesis
| Authors | Jian Han et al. |
| Year | 2026 |
| HF Upvotes | 15 |
| arXiv | 2604.13030 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
While diffusion models dominate the field of visual generation, they are computationally inefficient, applying a uniform computational effort regardless of different complexity. In contrast, autoregressive (AR) models are inherently complexity-aware, as evidenced by their variable likelihoods, but are often hindered by lossy discrete tokenization and error accumulation. In this work, we introduce Generative Refinement Networks (GRN), a next-generation visual synthesis paradigm to address these issues. At its core, GRN addresses the discrete tokenization bottleneck through a theoretically near-lossless Hierarchical Binary Quantization (HBQ), achieving a reconstruction quality comparable to continuous counterparts. Built upon HBQ's latent space, GRN fundamentally upgrades AR generation with a global refinement mechanism that progressively perfects and corrects artworks -- like a human artist painting. Besides, GRN integrates an entropy-guided sampling strategy, enabling complexity-aware, adaptive-step generation without compromising visual quality. On the ImageNet benchmark, GRN establishes new records in image reconstruction (0.56 rFID) and class-conditional image generation (1.81 gFID). We also scale GRN to more challenging text-to-image and text-to-video generation, delivering superior performance on an equivalent scale. We release all models and code to foster further research on GRN.
Engineering Breakdown
Plain English
This paper introduces Generative Refinement Networks (GRN), a new approach to image generation that combines the efficiency benefits of autoregressive models with the quality of diffusion models. The key innovation is Hierarchical Binary Quantization (HBQ), a near-lossless compression technique that solves the discrete tokenization problem that has historically limited autoregressive visual models. GRN achieves reconstruction quality comparable to continuous latent spaces while maintaining the complexity-awareness of autoregressive generation, where the model naturally allocates more computation to harder-to-generate regions. This addresses a fundamental inefficiency in current diffusion models, which apply uniform computational effort regardless of image complexity.
Core Technical Contribution
The core technical novelty is Hierarchical Binary Quantization (HBQ), a theoretically motivated quantization scheme that achieves near-lossless compression in discrete latent spaces where prior approaches suffered from lossy information loss. Unlike standard vector quantization used in VQ-VAE variants, HBQ uses a hierarchical binary tree structure to preserve fine-grained information across multiple levels of granularity. Built on this foundation, GRN reframes visual generation as a complexity-adaptive autoregressive process, where the model learns to allocate variable computation based on image difficulty rather than processing all regions uniformly like diffusion models. This represents a fundamental shift in how neural image generation trades off quality, speed, and computational efficiency.
How It Works
GRN operates in two main stages: encoding and generation. First, an image is encoded into a discrete latent space using HBQ, which progressively quantizes features through a binary hierarchy, preserving more information than traditional quantization at each level while remaining fully discrete. The hierarchical structure allows the model to reconstruct images with minimal information loss, comparable to continuous VAEs. During generation, an autoregressive transformer processes these tokens sequentially, but crucially, it learns to predict which regions require more tokens (higher complexity) and which can be represented sparsely. The model generates variable-length token sequences conditioned on image content and local context, allowing it to spend more computation on complex textures and fewer tokens on smooth regions. At inference, this leads to faster generation for simple images and more detailed generation for complex ones, all while maintaining discrete tokenization.
Production Impact
Adopting GRN would fundamentally change image generation pipelines by reducing inference latency while improving quality, particularly for batch-diverse workloads where image complexity varies. Currently, diffusion models require fixed computational steps (typically 20-50 diffusion iterations) per image regardless of content; GRN's variable-length generation means simple backgrounds might render in 30% of the tokens while detailed scenes use the full budget, reducing average latency and memory footprint. Integration requires replacing the VAE encoder/decoder with HBQ-based modules and swapping the diffusion backbone for an autoregressive transformer, which most teams already have infrastructure for. The main trade-off is that autoregressive generation is inherently sequential (harder to parallelize within a single image) whereas diffusion steps parallelize across the latent space, so throughput gains depend on batch sizes and whether you can exploit sequence-level parallelism. For real-time applications (video generation, interactive tools) and resource-constrained environments (mobile, edge), this could be immediately valuable; for batch rendering farms, the win is more subtle.
Limitations and When Not to Use This
The paper addresses discrete tokenization but doesn't fully solve error accumulation in autoregressive generation—mistakes in early tokens still propagate to later predictions, and this compounding effect may become problematic for very long sequences. The complexity-awareness property assumes that model uncertainty correlates well with visual complexity, which may break down for adversarial or out-of-distribution content; the paper likely doesn't extensively test robustness to distribution shift. HBQ's theoretical near-losslessness assumes sufficient hierarchy depth, but the compute/quality trade-off curve and optimal hierarchy design for different image types remain underexplored. The work is compared primarily against diffusion baselines; comparative analysis against other discrete AR methods (e.g., recent VQ-GAN variants, masked image modeling) appears incomplete, making it unclear if HBQ specifically is the breakthrough or if AR generation in general is the key factor.
Research Context
GRN builds on decades of autoregressive modeling (Transformer language models, PixelCNN) and recent advances in discrete image tokenization (VQ-VAE, VQ-GAN), attempting to reconcile why diffusion models achieved better quality despite autoregressive methods being theoretically sound. It positions itself against the current dominance of diffusion models (DDPM, Stable Diffusion, DALL-E 3) which have strong empirical results but inefficient uniform compute allocation. The paper opens a research direction around complexity-adaptive generation and hierarchical quantization, challenging the assumption that continuous latent spaces are necessary for high-quality synthesis. This work may inspire follow-up research into hybrid AR-diffusion approaches, better understanding of why discrete tokenization had been limiting, and more efficient generation paradigms for multimodal and video synthesis.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::