Pointer-CAD: Unifying B-Rep and Command Sequences via Pointer-based Edges & Faces Selection
| Authors | Dacheng Qi et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.04337 |
| Download | |
| Categories | cs.CV, cs.CL |
Abstract
Constructing computer-aided design (CAD) models is labor-intensive but essential for engineering and manufacturing. Recent advances in Large Language Models (LLMs) have inspired the LLM-based CAD generation by representing CAD as command sequences. But these methods struggle in practical scenarios because command sequence representation does not support entity selection (e.g. faces or edges), limiting its ability to support complex editing operations such as chamfer or fillet. Further, the discretization of a continuous variable during sketch and extrude operations may result in topological errors. To address these limitations, we present Pointer-CAD, a novel LLM-based CAD generation framework that leverages a pointer-based command sequence representation to explicitly incorporate the geometric information of B-rep models into sequential modeling. In particular, Pointer-CAD decomposes CAD model generation into steps, conditioning the generation of each subsequent step on both the textual description and the B-rep generated from previous steps. Whenever an operation requires the selection of a specific geometric entity, the LLM predicts a Pointer that selects the most feature-consistent candidate from the available set. Such a selection operation also reduces the quantization error in the command sequence-based representation. To support the training of Pointer-CAD, we develop a data annotation pipeline that produces expert-level natural language descriptions and apply it to build a dataset of approximately 575K CAD models. Extensive experimental results demonstrate that Pointer-CAD effectively supports the generation of complex geometric structures and reduces segmentation error to an extremely low level, achieving a significant improvement over prior command sequence methods, thereby significantly mitigating the topological inaccuracies introduced by quantization error.
Engineering Breakdown
Plain English
This paper addresses the problem of automatically generating CAD models using large language models, where prior work represents CAD operations as simple command sequences that fail to handle complex editing operations requiring entity selection (like filleting or chamfering). The authors introduce Pointer-CAD, a framework that represents CAD commands with explicit pointers to geometric entities (faces, edges) rather than just discrete command tokens, enabling the model to perform sophisticated geometric operations. This approach also handles the continuous variables in sketch and extrude operations more carefully to avoid topological errors. The pointer-based representation is a fundamental shift that makes LLM-based CAD generation practical for real engineering workflows.
Core Technical Contribution
The core innovation is replacing discrete CAD command sequences with a pointer-based representation that explicitly tracks and selects geometric entities during CAD construction. Rather than the model learning to generate uninterpreted command tokens, each command can now reference specific faces, edges, or other geometric primitives using learned pointers, similar to pointer networks but applied to 3D geometry manipulation. This enables the model to represent dependent operations—where later steps reference results of earlier steps—which is essential for realistic CAD workflows where you select a face and then apply an operation to it. The framework also addresses numerical stability by handling continuous parameters more carefully during discretization, reducing topological errors that plague sequence-only approaches.
How It Works
The system takes a natural language description of a CAD model as input and uses an LLM backbone to generate a sequence of commands, but critically, each command includes pointer outputs that index into a set of available geometric entities. As the model constructs the CAD model step-by-step, it maintains a geometry context—a representation of all current faces, edges, and vertices. When the model needs to perform an operation like 'chamfer the front edge,' the pointer mechanism selects which edge from the current geometry context to operate on, rather than relying on implicit positional encoding. Between major operations (sketch, extrude, etc.), the system updates the geometry context by extracting new entities created by the operation. For continuous parameters like extrusion distance or sketch coordinates, the model generates values that are more carefully discretized to preserve topological validity, potentially using learned bounds or quantization schemes that respect geometric constraints.
Production Impact
In production CAD generation pipelines, Pointer-CAD eliminates a major limitation: the inability to perform practical editing operations that practitioners use constantly (fillets, chamfers, draft angles, holes). This means workflows could shift from generating complete parts from scratch to generating rough structures and letting the model refine them with localized operations—a more incremental and controllable process. The explicit entity selection also makes the generated commands more interpretable and editable by humans, since you can see exactly which geometric feature is being modified rather than debugging implicit dependencies. However, this approach requires maintaining a dynamic geometry representation during inference, adding computational overhead compared to pure sequence generation; the system needs to track and update a 3D geometry context state, which could increase latency and memory usage in real-time applications. Integration would require coupling the LLM with a geometry backend (OpenCASCADE, Autodesk, etc.) rather than operating purely in token space, adding engineering complexity but enabling proper topological validation.
Limitations and When Not to Use This
The paper's approach assumes a well-behaved geometry context that can be reliably extracted and maintained throughout generation, but this breaks down with degenerate geometries, self-intersecting sketches, or malformed intermediate states—all common in real user workflows where people make mistakes and undo operations. The pointer mechanism introduces a new class of errors (selecting the wrong entity) that may be harder to recover from than simple token errors in sequence models, and the paper doesn't fully specify how to handle ambiguous selections or when multiple entities match a description. The method likely requires significant amounts of paired (natural language, CAD command sequence) training data annotated with entity selections, which doesn't exist at scale in the open domain—most CAD data lacks natural language descriptions, limiting generalization. Additionally, the discretization strategy for continuous parameters is mentioned but not fully detailed in the abstract, leaving unclear how well it actually handles the topological error problem or whether it simply moves the problem to a different representation.
Research Context
This work builds on the emerging trend of applying LLMs to code-like structured domains (Program Synthesis, SQL generation), but specializes the approach to the geometric domain where entity references matter. Prior work on LLM-based CAD (referenced implicitly) treated CAD as a pure sequence problem analogous to code generation, but Pointer-CAD recognizes that CAD is fundamentally different because operations are stateful and depend on selecting specific geometric objects. The research connects to broader work on pointer networks and retrieval-augmented generation, adapting those mechanisms to 3D geometry. It opens a research direction toward hybrid systems that combine symbolic geometric solvers with neural sequence models, potentially pointing toward 'neurosymbolic CAD' where the neural part handles high-level reasoning and the symbolic part handles geometric correctness guarantees.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::