Segmentation and semantics

A trained splat scene knows nothing about the objects inside it: every Gaussian is an anonymous ellipsoid, and “the sofa” or “that column” is a region a person points at, not something the data already labels. segmentationAssigning Gaussians in a trained scene a discrete object or category identity so a specific object can be selected, isolated or edited independently of the rest of the scene. Current methods derive these identities by lifting 2D masks, typically from the Segment Anything Model, into 3D, either as a whole-scene labelling pass or as a per-click prompted query. is the general name for fixing that, giving Gaussians a discrete object or category identity so a specific thing can be selected, isolated or edited on its own. It’s one of the more active research fronts in 3DGS right now, and, unlike the editors and compression formats covered elsewhere in this part, it hasn’t yet consolidated into something a mainstream tool ships as a feature.

Two shapes of the problem

The research splits into two working patterns. Segment Any 3D Gaussians (SAGA) attaches a scale-gated affinity feature to every Gaussian, distilled from the Segment Anything Model (SAM), so that a single 2D click prompt segments the corresponding 3D object in a few milliseconds without retraining the scene. It’s promptable and fast, but it answers one query at a time; it doesn’t hand back an inventory of every object in a scene up front. Gaussian Grouping takes the opposite shape: it augments every Gaussian with a compact identity encoding, supervised jointly during training from SAM masks tracked consistently across views plus a 3D spatial-consistency regulariser, producing a whole scene where every Gaussian already carries an object or “stuff” label. The cost is real: because the identity encoding is learned during training, Gaussian Grouping needs the original multi-view dataset, the source images and camera poses, not a bare exported PLY or SOG, and running it means re-training a labelled version of the scene rather than annotating an existing one.

Feature fields: querying instead of clicking

A third line skips discrete object IDs altogether and distils continuous features onto Gaussians instead. LangSplat trains a scene-specific autoencoder so that CLIP language features, plus SAM-derived hierarchical semantics, can be embedded per Gaussian and rendered efficiently, letting a scene answer an open-vocabulary text query like “the red chair” directly rather than requiring a stored label. A 2026 IEEE TPAMI survey covering the wider segmentation, editing and generation literature built on 3DGS confirms this is still a fragmented, fast-moving research area with several competing representations rather than one converged approach.

The practical gap this leaves

None of this is a checkbox in an editor yet. As of July 2026, Gaussian Grouping’s repository (lkeab/gaussian-grouping) has had no substantive commits since mid-2024, over a thousand stars notwithstanding, which is the profile of a stable reference implementation rather than a maintained tool: it still works, but it’s frozen where it was left. SAGA’s repository (Jumpat/SegAnyGAussians) is quieter still, last pushed in March 2025. Both are research code first, requiring their own environment and, for Gaussian Grouping, a full retraining run, not something dropped into an existing delivery pipeline over an afternoon.

For a scene that’s already been trained, compressed and delivered, the more workable pattern in practice is lighter-weight: run an open-vocabulary 2D object detector against a handful of rendered views of the existing scene, lift each 2D detection into 3D by checking which Gaussians actually sit at the right depth from multiple views rather than trusting one flat bounding box, and write the result into a companion per-Gaussian ID texture that a shader reads at render time to isolate or recolour the object. This avoids Gaussian Grouping’s retraining requirement entirely, works directly on an already-delivered SOG, and only costs work for the objects actually asked about rather than labelling the whole scene up front. It’s a narrower answer than “the scene is fully segmented,” but it’s the shape of answer that’s actually achievable against a production asset today.

In practice

Don’t expect a segmentation button in SuperSplat or splat-transform in 2026; there isn’t one, and the research covered here explains why it’s a harder feature than cropping or floater removal, it needs either a full retraining pass or its own inference pipeline layered on top of a delivered scene. If a project genuinely needs “click an object, do something to it,” budget it as a custom engineering task: either integrate Gaussian Grouping at training time, accepting that it wants the original capture dataset, or build a targeted 2D-detect-and-lift pipeline against the scene already in production. Treat any claim of turnkey splat object segmentation with real scepticism until it names which of these two costs it actually paid.