In a recent post, our CEO Ruben shared why constraint-driven AI matters and how it shifts the role of designers and engineers into true co-creation. That piece, along with my CDFAM interview, explored the pain points that inspired our work: non-negotiable geometry in automotive, architectural design lock-in, and the costly late-stage changes that plague hardware teams.
This post builds on that foundation. Instead of focusing on the “why,” here we’ll go deeper into the “how”: the workflow that makes constraint-aware 3D generative design practical, and what new possibilities it unlocks for designers and engineers.
The problem
Industrial design always starts with hard points—clearances, battery packs, safety envelopes, structural elements—non-negotiable geometry that must survive every iteration. Real creativity means exploring every viable form within that rigid framework.
Yet most generative-AI tools treat these guardrails as afterthoughts. Two issues show up again and again:
- Fiction before physics — Models produce striking shapes that collapse when confronted with packaging, safety, or structural data.
- Ambiguity in 2D guidance — When designers provide input through images, sketches, or edited renders, models often confuse whether the edit should update texture (paint, logos, finishes) or geometry (indents, curves, structural features). A highlight on a car body might be misread as a paint effect, when in reality it should change the shape.
Our approach
Instead of patching those gaps after the fact, we bake constraints directly into the generative process. Using Trellis-based 3D diffusion, masked generation, and differentiable rendering, we let you sweep through shapes, textures, and styles while critical geometry stays frozen. Crucially, our pipeline can explicitly separate texture edits from geometry edits—or let the AI resolve the distinction automatically based on multiple edited views.
From small-scale experiments—like a “bananacopter” wrapped around fixed screws and rotor blades that we’ll show later—to full automotive envelopes locked to battery packs and crash structures, the same workflow applies: critical geometry stays untouched while everything else evolves. Designers can add details with a few brushstrokes, lock down successful iterations, and push new variations—all without breaking the engineering rules in the background.
What this unlocks
- Constraint-aware generation — Freeze non-negotiable geometry while exploring new shapes, textures, and styles around it.
- Targeted iteration — Regenerate only in editable regions instead of starting over, keeping what already works.
- 2D-to-3D editing — Apply brushstrokes, sketches, or image edits to renders and propagate them back into the 3D model.
- Texture vs. geometry control — Decide explicitly whether an edit changes surface appearance or underlying shape—or let the AI resolve it from multiple views.
- Physics and style in sync — Bridge the “fiction-before-physics” gap so expressive designs remain viable against engineering constraints.
These capabilities matter because they address the biggest shortcomings in today’s generative 3D AI. To see why our approach is different, it helps to look at where the field stands today—and why standard models still fall short in real design workflows.
Generative 3D AI today
Over the past year, generative models for 3D objects have taken a big leap forward. The quality of single generations is now impressive—but for real design work, “one decent output” isn’t enough. In many cases, creatives spend as much time cleaning up assets as they would starting from scratch.
These limitations come from the models themselves. Because they are trained on large but biased datasets, they tend to favor certain shapes and styles. And while controllability has improved—mainly through image and text conditioning—these methods still leave designers with limited precision. They help nudge a model, but they don’t guarantee geometry that survives real-world constraints.
The baseline: image conditioning
Most 3D generative models, including Trellis, accept images as conditioning. In principle, you can draw edits on a render, feed that back into the model, and expect the change to appear in 3D. In practice, it rarely works reliably: surface details get lost, and it’s ambiguous whether an edit is meant to change texture (paint, logos) or geometry (indent, curve).
Image conditioning nudges a model, but it doesn’t guarantee precision. For real design work, that’s not control—it’s guesswork.
For example, if we sketch the letters DM on a render of a banana and use that as input, the resulting 3D generation often distorts or ignores the edit.
This lack of precision is one reason creatives hesitate to adopt 3D generative tools. It sets the stage for more robust methods—ones that don’t just nudge a model, but directly enforce constraints and interpret edits correctly.
Techniques that close the gap
The techniques we showcase here aim to bridge that gap: moving from nice-to-look-at outputs toward usable, constraint-respecting assets. In our demo, we show how to get precisely edited 3D assets that keep fixed geometry intact. Two key methods make this possible:
- Masked generation — borrowed from image inpainting/outpainting, this lets us freeze non-editable regions and regenerate only where new geometry is allowed.
- Differentiable rendering — a more advanced method where edits are made in 2D (on renders of the asset), then backpropagated through the renderer into the 3D latent space, updating either textures or geometry as intended.
We apply these ideas on top of the Trellis 3D generative model (see our Trellis tutorial blog post), though they can be used with other architectures as well. Trellis is especially useful because its structured latent space can decode into multiple modalities, including Gaussian splats and meshes—making it practical both for fast exploration and for downstream CAD or simulation. (It’s also widely adopted, with over 10K stars on GitHub.)
The bananacopter example
To illustrate, our demo walks through designing a toy helicopter around a fixed block with a screw and rotor blades. Using a 3D generative model with masked generation and differentiable rendering, we grow a whimsical “bananacopter” that respects the fixed geometry while still exploring creative variations. This simple example shows how the same workflow can scale to far more complex, constraint-heavy domains like automotive packaging.
Masked Generation
Masked generation is similar to image inpainting or outpainting. During denoising in the diffusion process, specified regions remain fixed. The idea was mentioned in the original Trellis paper under the name “Repaint,” although the official code was not released. There’s an unofficial implementation, but we found that a simpler approach works surprisingly well: initialize the generative voxel space with the constraints and regenerate only in masked regions outside of them. It’s straightforward to implement and pairs nicely with Trellis’ image conditioning.
By defining a mask to indicate which parts of the geometry the model is allowed to regenerate, and supplying guiding images, you can control both the 3D shape and texture. This gives designers creative flexibility while ensuring engineering-critical geometry stays untouched.
The example below shows constrained generation for the bananacopter: a toy helicopter wrapped around fixed screws and rotor blades. Only the free region is regenerated, guided by an image prompt, while the blades and gear block remain frozen.
Precise texture editing with differentiable rendering
To overcome the limits of image conditioning, we developed algorithms that use differentiable rendering to feed edits directly back into the model. Instead of hoping the network interprets a sketch correctly, this approach maintains a computational link between the rendered pixels and the structured latent representation of Trellis.
The process works like this: starting from a latent (from a prompt or a masked generation), we decode it into Gaussian splats and render multiple viewpoints with a differentiable renderer. When a designer edits those renders—for example, painting the letters DM on the banana—those pixel edits are backpropagated into the latent. Iteratively, the latent updates until its decoded output matches the edited views.
This makes edits much more precise than image conditioning alone, because it avoids the information loss that happens when neural networks “interpret” guidance. With just 3–4 edited renders, you can enforce detailed changes that persist from any viewpoint in 3D space.
Editing geometry through 2D input
Differentiable rendering doesn’t just work for textures—it can also drive geometry edits. Combined with masked generation, this lets us update the underlying shape of an object using only a few edited 2D renders.
The process starts with masked generation: fixed geometry is preserved, while new voxels are generated in editable regions. Instead of guiding this only with images, we augment the pipeline with a differentiable voxel renderer. This ensures that edits made in 2D renders—such as an indent or curve—propagate directly into the 3D structure.
In practice, this means designers can sketch structural changes in a couple of 2D views, and the model updates its geometry to match—without breaking constraints.
Differentiable rendering for geometry: manual edits in 2D renders (left) are backpropagated into the voxel latent, producing consistent 3D geometry updates (right).
Closing thoughts
Constraint-aware generation doesn’t just make 3D assets look good—it makes them usable. By freezing critical geometry and letting everything else stay fluid, the workflow you’ve seen here short-circuits the usual back-and-forth between design and engineering. The result is faster cycles, fewer dead ends, and assets that can move directly into CAD, simulation, or tooling.
But the bigger shift is what this unlocks: controllability. Designers can steer generation at the right level of detail—whether that’s broad form exploration or pixel-level edits—while staying aligned with engineering constraints. And because these methods are modular, they don’t just live in demos: we can integrate them into existing graphical pipelines, connect them to product workflows, and adapt them across industries.
If you’re working on design tools, simulation, or automation pipelines and this resonates, we’d love to connect. Reach out to explore how our constraint-driven generative pipeline can streamline your projects—or spark new ones.