Differentiable rasterisation

Training a splat scene means running the same rasterisationConverting geometric primitives directly into pixels on screen, rather than tracing or marching rays through a scene. 3D Gaussian Splatting adapts a tile-based GPU rasterisation pipeline to render Gaussians instead of triangles. pipeline described in how splats render, project, sort, blend, but with every step built so it can be run backwards: given how wrong a rendered pixel is, work out how each Gaussian’s position, shape, opacity and colour should change to make it less wrong. That backward pass, not the forward render, is where the actual optimisation happens, and its design is what makes training millions of Gaussians on a single GPU practical at all.

Forward: build up a value you can differentiate

The forward pass renders exactly as described in the rendering pipeline article: each Gaussian projects to a 2D ellipse via a Jacobian approximation of the camera’s projection, Σ' = J W Σ Wᵀ Jᵀ where W is the view transform, and the accumulated pixel colour is the ordered alpha blendingA compositing technique that accumulates colour from ordered, semi-transparent primitives, each weighted by its own opacity and by how much light nearer primitives have already absorbed. Splat rasterisers use it to blend overlapping Gaussians into a final pixel colour. sum of every Gaussian that overlaps that pixel, weighted by opacity and by how much colour nearer Gaussians have already absorbed. The loss function compares that accumulated colour against the training photograph, using a weighted mix of L1 pixel error and a structural similarity term, D-SSIM. Every operation in that chain, projection, covariance transform, alpha compositing, the loss comparison itself, is a plain differentiable function, so in principle you could hand the whole thing to an automatic-differentiation framework and get gradients for free.

Why the naive backward pass doesn’t scale

In practice, a straightforward autodiff implementation would need to remember, for every pixel, the ordered list of every Gaussian that contributed to it and the running opacity accumulated at each step along that list, so it can walk back through the blend afterwards. Some pixels under dense geometry might be touched by hundreds of overlapping Gaussians; that’s an arbitrarily long, per-pixel, variable-length buffer, which is a poor fit for fixed-size GPU memory and kills throughput at exactly the scale, millions of Gaussians, that makes this method worth using. The 3D Gaussian Splatting paper’s rasteriser avoids storing that list at all.

The trick: recompute the blend state instead of storing it

The forward pass keeps only the final accumulated opacity for each pixel, nothing about the intermediate steps that produced it. The backward pass re-traverses each tile’s sorted Gaussian list a second time, this time back-to-front, and recovers the running opacity at each step algebraically: dividing the final accumulated value by each Gaussian’s own contribution as it’s encountered peels the blend apart one layer at a time, back to front, without ever having stored the intermediate states during the forward pass. Screen space stays tiled at the same 16x16 resolution used for sorting, and a Gaussian’s contribution to the gradient stops being computed once its depth is behind the last Gaussian that actually still contributed colour at that pixel in the forward pass, so fully-occluded geometry costs nothing on the way back either. The result is a backward pass with the same tile-parallel structure as the forward one, using roughly the same amount of stored state, rather than one that grows with scene overlap.

From a pixel error back to a quaternion

Once a gradient exists for a pixel’s accumulated colour, the chain rule carries it back through every stage covered in the earlier articles. It flows through the alpha-blend weights to each contributing Gaussian’s opacity and spherical-harmonics colour coefficients; through the 2D covariance to the projection Jacobian and back to the 3D covariance; and through the Σ = R S Sᵀ Rᵀ decomposition covered in 3D Gaussians and covariance to the underlying scale vector and rotation quaternion specifically, rather than to the covariance matrix as a monolithic object. That last step matters: because scale and rotation are optimised directly, gradient descentAn optimisation method that repeatedly nudges parameters in the direction that most reduces a loss function, following that loss's gradient with respect to each parameter. Splat training uses it, via backpropagation through the differentiable rasteriser, to adjust every Gaussian's position, shape, opacity and colour. updates can move a Gaussian’s shape and orientation independently without ever risking an invalid, non-positive-semi-definite matrix, the same constraint problem that motivated the parametrisation in the first place. Position, opacity and colour each accumulate their own gradients from every pixel and every training view they contributed to, then get updated by Adam, a common adaptive variant of stochastic gradient descent, with the position parameter using its own decaying learning-rate schedule because its gradients naturally sit at a very different scale from opacity’s or colour’s.

Why this is what makes training real-time-renderable scenes possible

None of this involves a neural network anywhere in the loop. The optimisation target is a fixed, explicit set of Gaussian parameters, and both the forward render and the backward gradient pass are the same tile-based rasterisation workload, just run in opposite directions. That symmetry is what lets a scene with several million Gaussians train in minutes on a single consumer GPU rather than the hours a comparable NeRF needed: there’s no per-sample network forward-and-backward pass hiding inside every training step, only geometric projection and compositing math that a rasteriser was already built to do fast.

In practice

You won’t touch this pipeline directly from PostShot or PlayCanvas, but its shape explains two things worth knowing. First, training cost scales with how many pixels and Gaussians actually overlap, not with scene complexity in the abstract, which is why an over-dense, unpruned region slows training down as well as bloating file size, the backward pass has to walk every one of those tile lists twice. Second, because gradients only exist for Gaussians a training view actually rendered, any region your capture never looked at from a useful angle gets no meaningful gradient signal at all; it stays whatever the initial COLMAP point and default parameters gave it, which is a mathematical reason, not just a heuristic one, for why capture coverage gaps show up as genuinely unconstrained geometry rather than merely blurry geometry.