3D Gaussians and covariance

Every primitive in a splat scene is a 3D Gaussian: a fuzzy ellipsoid with a centre, a size, an orientation, an opacity and a colour. The centre is easy. The rest of the shape lives in a single mathematical object, the covariance matrixA matrix describing a Gaussian's spread and orientation in space: its terms define how far the ellipsoid extends along each axis and how those axes are tilted, together encoding size, shape and orientation as a single object., and how that object gets parametrised is the reason splat training is stable enough to work at all. This is the base the other maths topics in this section build on: spherical harmonics, differentiable rasterisation and densification all operate on a Gaussian defined this way.

Mean: just a position

The mean of a 3D Gaussian is nothing more than an x, y, z position in world space, the same as a point in your COLMAP sparse point cloud. If a Gaussian had only a mean and no covariance, you’d have a point cloud, not a splat scene. Every point PostShot’s initial reconstruction gives the optimiser becomes exactly one Gaussian’s mean, which is also why a thin or noisy COLMAP point cloud in a region tends to under-reconstruct that region until densification fills it in later in training.

Covariance: size and orientation in one object

A covariance matrix generalises “radius” into something that can be stretched and rotated. Instead of one number describing how big a blob is, it describes how far it extends along each axis and how those axes are tilted relative to world space. Geometrically it’s the definition of an ellipsoid: feed it into a Gaussian’s density function and the result is a cloud of probability mass that’s dense at the centre and falls off faster or slower depending on direction. A large, thin, tilted Gaussian and a small, round one both come from the same kind of matrix, just with different numbers in it.

Why not store the matrix directly

You’d think the natural thing to optimise is the covariance matrix itself, but a covariance matrix has a hard constraint: it must be positive semi-definite, otherwise it doesn’t correspond to any real ellipsoid at all. Plain gradient descent has no way to guarantee that constraint holds after an update; nudge the wrong entry the wrong way and you get a matrix that describes nothing. The 3D Gaussian Splatting paper sidesteps this by never touching the matrix directly. Instead each Gaussian stores a 3D scale vector and a rotation, expressed as a quaternion, and the covariance is reconstructed each time as Σ = R S Sᵀ Rᵀ, where S is the diagonal scale matrix and R is the rotation matrix built from the quaternion. Multiplying a matrix by its own transpose always produces something positive semi-definite, so the construction makes the constraint impossible to violate rather than something the optimiser has to respect. Scale and rotation are also just more useful to work with: they’re the same handles a 3D artist already reasons about, and gradient descent can push on them independently without needing to know anything about the constraint underneath.

Anisotropy: why splats aren’t spheres

A Gaussian whose scale is equal along all three axes is isotropic: a sphere, the same in every direction. Real surfaces are not spheres, and neither is almost anything a splat needs to represent efficiently. An anisotropicHaving properties that differ by direction, as opposed to isotropic (the same in every direction). An anisotropic Gaussian's covariance can stretch it into a thin disc or needle to match a surface, rather than staying a uniform sphere. Gaussian, one with different scale along different axes, can flatten into a thin disc that hugs a wall or a floor, or stretch into a needle that follows a wire or a blade of grass. This is what lets a splat scene represent a flat surface with a handful of large, thin ellipsoids instead of thousands of small round ones: anisotropy converts shape information that a point cloud has to fake with density into something a single primitive can encode directly. It’s also the direct descendant of a much older idea: elliptical, anisotropic Gaussian kernels were already the basis of EWA volume splatting, the volume-data variant of the EWA framework Zwicker et al. published alongside the surface-oriented Surface Splatting in 2001, decades before radiance fields existed, and 3D Gaussian Splatting reuses that same elliptical-footprint reasoning for projecting Gaussians onto the screen.

In practice

You’ll never edit a covariance matrix by hand, but its consequences show up constantly. A splat that looks like it’s made of needles or paper-thin shards, rather than solid volume, usually has Gaussians whose scale has grown too anisotropic in one axis relative to the others, often in a region the training views never covered from enough angles to constrain a genuine 3D shape. Sparse or single-direction capture passes are the most common cause: COLMAP has no evidence for depth along the missing viewing direction, so the optimiser is free to stretch a Gaussian arbitrarily thin along it and still fit the training photos. When you’re planning capture coverage for a scene with thin structures, railings, foliage, glass, this is the underlying reason extra oblique angles matter: they’re constraining covariance, not just filling in colour.