SurgFormer: Scalable Learning of Organ Deformation with Resection Support and Real-Time Inference
| Authors | Ashkan Shahbazi et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.06543 |
| Download | |
| Categories | cs.CV |
Abstract
We introduce SurgFormer, a multiresolution gated transformer for data driven soft tissue simulation on volumetric meshes. High fidelity biomechanical solvers are often too costly for interactive use, so we train SurgFormer on solver generated data to predict nodewise displacement fields at near real time rates. SurgFormer builds a fixed mesh hierarchy and applies repeated multibranch blocks that combine local message passing, coarse global self attention, and pointwise feedforward updates, fused by learned per node, per channel gates to adaptively integrate local and long range information while remaining scalable on large meshes. For cut conditioned simulation, resection information is encoded as a learned cut embedding and provided as an additional input, enabling a unified model for both standard deformation prediction and topology altering cases. We also introduce two surgical simulation datasets generated under a unified protocol with XFEM based supervision: a cholecystectomy resection dataset and an appendectomy manipulation and resection dataset with cut and uncut cases. To our knowledge, this is the first learned volumetric surrogate setting to study XFEM supervised cut conditioned deformation within the same volumetric pipeline as standard deformation prediction. Across diverse baselines, SurgFormer achieves strong accuracy with favorable efficiency, making it a practical backbone for both tasks. {Code, data, and project page: \href{https://mint-vu.github.io/SurgFormer/}{available here}}
Engineering Breakdown
Plain English
SurgFormer is a transformer-based neural network designed to predict how soft tissue deforms during surgical procedures in real-time, trained on data from expensive physics simulators. The key innovation is a multiresolution architecture that combines local message passing with global self-attention through learned gating mechanisms, allowing it to handle large volumetric meshes while remaining computationally practical. For surgical cuts, the model accepts encoded resection information as input, enabling a single unified model to handle both intact and modified tissue geometries. This approach brings interactive performance (near real-time rates) to biomechanical simulation, which is typically constrained to offline analysis due to computational cost.
Core Technical Contribution
The core novelty is the multiresolution gated transformer architecture that adaptively fuses local and global information using per-node, per-channel learned gates. Unlike standard transformers that either apply expensive global attention everywhere or sacrifice modeling capacity with purely local operations, SurgFormer builds a fixed mesh hierarchy and strategically applies coarse global self-attention at reduced resolutions while maintaining detailed local message passing at full resolution. The gating mechanism learns when to rely on nearby node interactions versus distant dependencies, effectively creating a learned, input-adaptive routing strategy. Additionally, the framework's unified handling of cut-conditioned simulation through learned cut embeddings represents a departure from prior approaches that required separate models or complex conditional logic.
How It Works
SurgFormer takes as input a volumetric mesh representing tissue geometry, nodewise initial conditions, and optionally a learned embedding representing surgical cuts or resections. The architecture builds a fixed hierarchical mesh structure with multiple resolution levels, enabling efficient processing of large meshes by focusing expensive global computation on coarser representations. During forward inference, the model applies repeated multibranch blocks consisting of three parallel pathways: (1) local message passing that updates node features based on neighboring nodes in the original mesh, (2) coarse global self-attention computed on the downsampled mesh hierarchy to capture long-range biomechanical effects, and (3) pointwise feedforward networks for nonlinear feature transformation. These three pathways are combined using learned per-node, per-channel gate values that weight each pathway's contribution, allowing the model to dynamically determine the relative importance of local versus global information for each node. The final output is a nodewise displacement field predicting how each mesh vertex moves under the simulated forces or tissue deformation.
Production Impact
This approach directly addresses the latency bottleneck in surgical simulation systems: traditional finite element method (FEM) solvers require seconds to minutes per time step, while SurgFormer inference runs at near real-time rates (exact latency not specified in abstract but implied to be orders of magnitude faster). For surgical planning and intraoperative guidance systems, this enables interactive feedback during procedure rehearsal or live execution, transforming simulation from an offline planning tool into an interactive assistant. The unified model for handling both intact and cut tissue eliminates the need to maintain multiple specialized models or implement complex conditional branching logic in production pipelines. However, the trade-offs include a dependency on high-quality training data from physics solvers (data generation cost upfront), potential distribution shift when real tissue properties differ from training data, and the need to maintain and inference a large transformer model, which still requires GPU acceleration for true interactivity on large meshes.
Limitations and When Not to Use This
SurgFormer's accuracy and generalization depend entirely on the quality and diversity of the solver-generated training data, meaning systematic biases in the underlying physics simulator will be learned and reproduced. The paper does not address how well the model generalizes to tissue properties, geometries, or boundary conditions significantly different from training examples—this is a critical limitation for heterogeneous patient anatomy in real surgery. The fixed mesh hierarchy means the model cannot adapt to dynamic remeshing or topological changes during surgery beyond the specific resection patterns it was trained on, limiting its applicability to complex multi-stage procedures. Additionally, the paper does not discuss failure mode analysis, uncertainty quantification, or how to detect when the model's prediction has drifted unacceptably from physics, which is essential for clinical adoption where simulation errors could affect surgical decisions.
Research Context
This work builds on a large body of research combining neural networks with physics simulation for accelerating expensive solvers, including prior work on graph neural networks for mesh-based simulation and transformer architectures for spatiotemporal prediction. The multiresolution gating idea extends concepts from hierarchical attention mechanisms and adaptive computation, previously explored in models like Swin Transformers and conditional gating networks, but applies them specifically to the biomechanical simulation domain with a focus on mesh-based volumetric data. The contribution fits a broader research direction toward learning-based surrogates for physics simulation that can trade some accuracy for dramatic speedup gains, particularly relevant to time-sensitive applications like interactive surgery. If successful, this work could inspire similar multiresolution gating approaches in other spatiotemporal prediction tasks where capturing both local and global dependencies at different computational costs is important (e.g., climate modeling, fluid dynamics, materials science).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::