SfM fundamentals

Before COLMAP writes a single camera pose, it has to solve a chicken-and-egg problem: it doesn’t know where the cameras were, and it doesn’t know where anything in the scene is, and it has to recover both at once from nothing but a folder of photos. SfMStructure-from-Motion: recovering camera poses and a sparse 3D point cloud simultaneously from a set of overlapping photographs, using matched features, epipolar geometry and triangulation. COLMAP's incremental mapper is the SfM implementation most splat pipelines rely on. is the family of techniques that does this, and the four ideas in this article, features, matching, epipolar geometry and triangulation, plus the bundle adjustmentThe global optimisation step that jointly refines every estimated camera pose and every triangulated 3D point to minimise total reprojection error. COLMAP re-runs it repeatedly during incremental reconstruction as new images register. step that ties them together, are what every COLMAP run is actually doing underneath the progress bar.

Features: giving the algorithm something to grab onto

The first step finds points in each image that are distinctive enough to recognise again from a different angle, different lighting, or a different scale, a window corner, a patterned tile edge, a sharp shadow line, and describes each one with a numeric fingerprint. COLMAP’s default is SIFT, a descriptor designed specifically to survive rotation, scale change and moderate lighting shift, though newer learned detectors are increasingly available as alternatives in current SfM toolchains. What actually matters for a practitioner is simpler than the algorithm: feature matchingComparing feature descriptors across images to propose which points likely correspond to the same physical location, prior to geometric verification. COLMAP supports exhaustive, sequential, vocabulary-tree, spatial and transitive matching strategies. downstream can only work with what feature extraction finds, and a plain wall, a clear sky or a glossy, reflective surface hands the extractor nothing to describe, no matter how good the descriptor is.

Matching: finding the same point twice

Once every image has its own set of described points, feature matching compares those descriptors across image pairs to propose which points are probably the same physical spot seen twice. This step is deliberately generous with false positives at first: a nearest-descriptor match is just a hypothesis, not a fact, and plenty of hypotheses are wrong, two different corners that happen to look similar, a repeated pattern that fools the descriptor, a reflection that isn’t really there at all. Raw matching output is noisy by design, which is exactly what the next step exists to filter.

Epipolar geometry: the geometric filter

epipolar geometryThe geometric constraint that, given two calibrated views of a rigid scene, a point's match in one image must lie along a specific line in the other image. COLMAP uses it to verify proposed feature matches and reject geometrically inconsistent ones. is the constraint that any two calibrated views of the same rigid scene must obey: for a point in one image, its match in the other image is restricted to lie along a specific line, not anywhere in the frame. COLMAP estimates this relationship for each image pair (as an essential or fundamental matrix, depending on whether calibration is known) and then throws out every proposed match that doesn’t obey it. This is where most of the false matches from the previous step die. It’s also a strict, purely geometric test: it doesn’t know or care what the scene contains, only whether the proposed correspondence is consistent with two cameras looking at one rigid 3D world, which is precisely why it’s so effective at separating real matches from coincidental ones.

Triangulation: turning two rays into a point

Once a pair of matched, geometrically-verified points exists and both cameras’ positions and orientations are known (or estimated), the two rays cast from each camera through its respective image point should meet at the 3D point that produced both observations: triangulationComputing a 3D point's position from two or more matched, pose-known camera views by finding where the rays cast through each observation intersect, or pass closest to each other once pose and pixel noise are accounted for. is the computation that finds that intersection, and in practice, because pose estimates and pixel locations both carry small errors, the two rays usually pass near each other rather than through the same point exactly, so the result is the closest point to both rather than a literal intersection. This is also why parallax, real separation between the two camera positions, matters so much: rays cast from two nearly identical viewpoints intersect at a shallow, poorly-defined angle, and small pixel noise turns into large depth uncertainty.

Bundle adjustment: fixing everything at once

Every individual triangulated point and every individual camera pose carries error, and those errors interact: a slightly wrong pose distorts every point it helped triangulate, and a slightly wrong point distorts every future pose estimated from it. Bundle adjustment is the global optimisation that adjusts every camera pose and every 3D point together to minimise total reprojection error, the summed distance between where each 3D point’s projection actually lands in each image and where it was originally observed. COLMAP runs this repeatedly through an incremental reconstruction, not just once at the end, registering a new image, triangulating new points, then re-running bundle adjustment to let the new information correct everything estimated so far, which is why reconstruction quality tends to improve as more overlapping images join the model rather than only as a final polish step.

In practice

None of this is COLMAP-specific: it’s the general SfM pipeline that COLMAP, and most tools that produce camera poses for splat training, implement in some form. What’s worth carrying into the next articles is the causal chain: weak features lead to weak matches, weak matches that survive the epipolar filter lead to badly-conditioned triangulation, and badly-conditioned triangulation leads to a bundle adjustment that either diverges or quietly settles on a distorted scene. When a reconstruction looks wrong, it’s worth asking which of these five stages actually broke, because the fix looks completely different depending on the answer, more overlap for weak matching, a different matcher for repetitive structure, or manual pair selection for a scene that never triangulated a confident initial pair.