Can Natural Image Autoencoders Compactly Tokenize fMRI Volumes for Long-Range Dynamics Modeling?
| Authors | Peter Yongho Kim et al. |
| Year | 2026 |
| HF Upvotes | 6 |
| arXiv | 2604.03619 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Modeling long-range spatiotemporal dynamics in functional Magnetic Resonance Imaging (fMRI) remains a key challenge due to the high dimensionality of the four-dimensional signals. Prior voxel-based models, although demonstrating excellent performance and interpretation capabilities, are constrained by prohibitive memory demands and thus can only capture limited temporal windows. To address this, we propose TABLeT (Two-dimensionally Autoencoded Brain Latent Transformer), a novel approach that tokenizes fMRI volumes using a pre-trained 2D natural image autoencoder. Each 3D fMRI volume is compressed into a compact set of continuous tokens, enabling long-sequence modeling with a simple Transformer encoder with limited VRAM. Across large-scale benchmarks including the UK-Biobank (UKB), Human Connectome Project (HCP), and ADHD-200 datasets, TABLeT outperforms existing models in multiple tasks, while demonstrating substantial gains in computational and memory efficiency over the state-of-the-art voxel-based method given the same input. Furthermore, we develop a self-supervised masked token modeling approach to pre-train TABLeT, which improves the model's performance for various downstream tasks. Our findings suggest a promising approach for scalable and interpretable spatiotemporal modeling of brain activity. Our code is available at https://github.com/beotborry/TABLeT.
Engineering Breakdown
Plain English
This paper addresses a fundamental bottleneck in fMRI analysis: modeling long temporal sequences of 4D brain imaging data while staying within GPU memory constraints. The authors propose TABLeT, which compresses 3D fMRI volumes into compact continuous tokens using a pre-trained 2D image autoencoder, then runs a standard Transformer encoder on these compressed sequences. This approach enables modeling significantly longer temporal windows than prior voxel-based methods while maintaining interpretability and performance, evaluated on large-scale datasets like the UK Biobank.
Core Technical Contribution
The core innovation is a two-stage tokenization strategy that reuses pre-trained 2D image autoencoders designed for natural images to compress 3D medical imaging volumes into a manageable token sequence. Rather than operating directly on millions of voxels (which causes memory explosion), the method treats each 3D fMRI volume slice-by-slice using a 2D encoder, producing a compact latent representation that preserves spatiotemporal information while reducing dimensionality by orders of magnitude. This allows a simple Transformer to capture long-range dependencies in brain dynamics without specialized medical-domain pre-training. The technique is orthogonal to existing fMRI modeling approaches and demonstrates that generic vision encoders can effectively compress medical imaging modalities.
How It Works
Input is a time series of 3D fMRI volumes, each containing thousands of voxels across three spatial dimensions. For each 3D volume, the method applies a pre-trained 2D autoencoder (e.g., trained on natural images) to encode the volume into a set of continuous tokens—essentially creating a learned, compressed representation of the brain activity pattern. These token sequences from consecutive timepoints form a long sequence that a standard Transformer encoder processes with self-attention, learning temporal dependencies across frames. The encoder produces contextualized representations of brain states at each timepoint, which can be fed to downstream tasks (classification, prediction, etc.). Critically, the 2D autoencoder runs efficiently on GPUs because it operates on individual 2D slices rather than the full 3D volume, and the resulting token counts are orders of magnitude smaller than raw voxel counts, enabling longer temporal windows and lower VRAM consumption.
Production Impact
Engineers building clinical fMRI analysis pipelines would gain a practical pathway to deploying long-context brain imaging models on standard hardware (8-16GB GPUs) that previously required expensive cluster setups. The approach is modular: you can swap pre-trained image autoencoders (e.g., different Vision Transformer backbones, different compression ratios) without retraining, allowing rapid experimentation and domain adaptation. Memory costs drop dramatically—processing 10-minute fMRI scans becomes feasible where processing 2-minute windows was the prior limit—opening up new clinical applications like detecting subtle temporal patterns in seizure or stroke recovery. However, you introduce a dependency on the quality of the pre-trained autoencoder and add an extra encoding step (latency trade-off), and the method assumes the 2D autoencoder can meaningfully compress fMRI without losing disease-relevant signal, which requires validation for each new task.
Limitations and When Not to Use This
The paper's success depends entirely on the assumption that pre-trained 2D vision autoencoders, trained on natural images, preserve fMRI-relevant information—this may break for rare pathologies or subtle biomarkers that differ dramatically from natural image statistics. The approach sacrifices some interpretability compared to voxel-space models because the token space is learned from image data, not neuroscience principles, making it harder to map findings back to anatomical brain regions without additional post-hoc analysis. The method is not tested for robustness to domain shift (different MRI scanners, acquisition protocols, populations), and scaling to ultra-long sequences (hours of continuous recording) may still hit Transformer quadratic attention complexity limits. There is no discussion of how to handle missing data, motion artifacts, or how the choice of pre-trained autoencoder affects performance across different downstream tasks.
Research Context
This work builds on the recent success of vision transformers and autoencoder-based compression in medical imaging, but applies it to the temporal dimension of fMRI—a domain where most prior work focused on spatial autoencoders or recurrent models. It extends the practical applicability of Transformer-based brain modeling beyond small-scale datasets by reducing memory footprint, enabling evaluation on large cohorts like UK Biobank. The paper opens a research direction around cross-domain transfer for medical imaging: can generic vision models, trained on millions of natural images, become universal feature extractors for diverse medical modalities? This aligns with broader trends in foundation models for medical imaging and suggests that domain-specific pre-training may not be necessary if compression is done intelligently.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::