Real-time photogrammetric stitching of high resolution video on COTS hardware

It is often necessary to stitch images together into a larger image; examples include generating a panoramic image, building an aerial terrain photo, and wide area surveillance. The latter is an example in which this stitching must happen in real time, i.e. less than one frame period. If further constraints are placed on the system such as operation in poor contrast conditions and multiple targets moving against a changing background in the system´s field of view, yet it must still operate deterministically, then current predominant methods using image registration do not suffice. These methods typically require a large overlap between adjacent cameras in a static staring array, and involve manipulating the individual video frame images by means of rotation, translation and scaling until detected feature points optimally match. This is non-ideal because more cameras are needed due to the overlap of the adjacent cameras, the requirement to determine salient image points (which may not be evident in poor contrast conditions), the iterative manner used to align images´ salient points, the frequent neglecting of lens distortion effects and the inability to align two cameras if the one between them fails. Photogrammetric methods have been used before to rectify these short comings, but have always required expensive, bulky, custom processing hardware. This paper shows that it is possible to stitch an array of minimally overlapped high resolution wide angle cameras irrespective of scene content and lighting changes in a deterministic manner via novel use of commercial graphics processing hardware and a parallelized algorithm.