A relative frame representation for fixed-time bundle adjustment in SFM

A successful approach in the recovery of video- rate structure from motion is to allow the camera to keep track of its position in every frame assuming the recovered set of scene landmarks is fixed in 3D, and then to use the poses in a subset of separated frames, or keyframes, to initialise further landmark structure. The landmark structure and keyframe poses are optimised in a bundle adjustment. Unfortunately this monolithic bundle adjustment has cubic complexity. This paper shows how representing landmarks and camera poses in relative frames, and by temporarily removing certain measurements, introduces a conditional indepedence which allows the bundle adjustment to be split into two parts. One "local" part involves the most recent keyframes and associated landmarks, and runs in constant time. The other "global" part deals with older keyframes and structure, and runs, as ever, in cubic time. Three important outcomes are: (i) the fixed- time local adjustment allows exploratory map-building to keep pace with camera pose tracking; (ii) it produces statistically consistent results; and (iii) referencing to relative frames means that any update in positions from the global adjustment are immediately incorporated in the local fixed-time adjustment. The relative frame approach is applied to the parallel tracking and mapping method for structure from motion, and its results shown to be identical, and the exploratory map building phase shown to maintain fixed time performance.