position estimation of mobile mapping imaging sensors using aerial imagery
Phillipp Fanta-Jende is a PhD student in the Department of Earth Observation Science (EOS). His supervisor is prof.dr.ir. M.G. Vosselman from the Faculty of Geo-Information Science and Earth Observation.
Mobile mapping has become an important extension to traditional geo-information acquisition techniques. Its unique ability to record street-level data using laser scanners or cameras on a large scale complements and augments the photogrammetric portfolio. Particularly in urban areas, MM plays a significant role, as it enables complementary data representations of the scene in conjunction with other data-capture solutions.
In general, urban areas pose a challenge with respect to satellite-based positioning solutions due to high-rise buildings and other tall structures in built-up areas which may obstruct the direct-line-of-sight to navigation satellites. As a consequence, multipath and non-line-of-sight effects may occur and impede the position estimation of the receiver.
Although MM platforms usually utilise inertial navigation to bridge potential signal outages (i.e. non-line-of-sight), a complete mitigation of these effects remains an unsolved problem, as the discrimination between a valid and an invalid signal (i.e. multipath) from navigation satellites is per se not possible. Hence, the accuracy of the MM platform’s position and thus its acquired data is unknown and likely impaired in such scenarios. Although the traditional approach to introduce ground truth into the correction procedure by using ground control points has been proven reliable and exact, the acquisition of such information is costly and labour-intensive. In the light of a growing market and a diversification of geo-data acquisition platforms, automated, cost-efficient, and scalable solutions are greatly demanded.
To this end, the present research investigates the possibility to integrate external ground truth derived from aerial images into the position estimation of the MM platform in order (1) to verify and (2) to improve its accuracy if possible.
Aerial images are a standard product in many countries and are acquired at regular intervals at a nationwide scale. The effects which hinder a reliable position estimation of MM platforms are not applicable to aircrafts, as the direct line-of-sight to navigation satellites is usually not obstructed. Calibrated cameras, highly accurate inertial sensors, and positioning equipment enable precise sensor orientation and thus the recording of high-resolution nadir as well as oblique imagery with great accuracy.
In order to utilise aerial images for the verification and improvement of MM data, both data sets need to be registered first. The MM data used for this research project are panoramic images, which are recorded in a discrete manner along the platform’s trajectory. Since the aerial and the MM image data set do not share the same perspective on the scene, many characteristics between the images are different and have a significant influence on the registration quality. This poses the main challenge of the present work.
The first chapter outlines the background, the research problem and objectives, the second chapter of the thesis focuses on exploring different techniques for a co-registration of aerial nadir and MM images. The first part of chapter two investigates the possibility to use low-level tie features for the registration task between aerial nadir and MM panoramic images. In order to simplify the correspondence problem, MM images are re-projected on an artificial ground surface, i.e. to simulate a top-down view similar to the aerial nadir image counterpart. The actual registration is based on low-level feature matching, which can be coarsely separated in a feature detection and description phase. To this end, different feature matching methods are compared with each other based on their performance to identify and register image features that are visible in both – the MM and the aerial nadir – data sets. Conducted experiments indicate that correspondences can be mostly identified at road markings. As road markings may be repetitive (e.g. zebra crossings), their feature descriptions can be ambiguous, which consequently leads to wrong correspondences.
These findings lead to a number of considerations, which are discussed in the second part of chapter two. Although MM images have an unknown accuracy, a certain margin of error can be assumed. To this end, the orientation parameters of the MM images are used to constrain the search space for correspondences. In order to exploit this property, orientation parameters are employed in conjunction with template matching. A kd-tree[1] is used to identify isolated points in order to avoid problems with repeated patterns in the MM image. These points are then projected into the aerial image using the MM orientation parameters. The window size of the template matching accounts for possible localisation errors of the platform while constraining the search for correspondences. Subsequently, these initial correspondences are used to compute a transformation between the image pair to reliably map the remaining (and possibly cluttered) points from one image into the other to enable an accurate registration. The experiments ascertain great improvements of the inlier rate compared to the previous approach. In certain scenarios, however, repeated patterns still cause ambiguities. Moreover, other differences between the images, e.g. illumination, contrast, and content may influence the performance of the registration procedure.
The third chapter outlines the development of an automatic and reliable co-registration procedure between aerial nadir and MM images for the adjustment of MM data. Based on previous insights, the procedure is further developed with respect to automated processing and to account for the overall image differences between aerial and MM images. This is achieved by utilising MM orientation parameters for the computation of an initial transformation which is verified for plausibility. Subsequently, this transformation is used to project salient corner points from the MM image into the aerial image. The exact correspondence is identified by using the phase information of image templates. With an inlier rate of about 98%, the co-registration technique proves to be successful. The image observations of the correspondences in the aerial nadir images are triangulated and used for the adjustment of the MM data. This approach is able to verify as well as improve the horizontal error of the MM data up to a decimetre. The vertical dimension, however, is prone to be worsened by this approach, as the aerial images have been acquired at a high altitude that potentially leads to an unfavourable intersection geometry.
Although aerial nadir and MM images have a different perspective on the scene, overlapping areas can be found mostly on the ground where road markings and other ground-based features can be identified as valid correspondences. Since not all roads have road markings or other distinct features on the ground which can be easily registered, the fourth chapter focusses on exploring the possibility to register MM images with aerial oblique images. Similar to the previous registration case, aerial oblique images do not share the same perspective on the scene with MM images. Mutual entities between both data sets are, however, façades and other vertical surfaces along the road. In order to utilise these areas for registration, though, another reprojection of the MM images is required. Unlike the ground, façade planes cannot be easily assumed. Hence, a sparse point cloud from MM images is generated to fit planes into the scene in order to create façade hypotheses. These surfaces can then be discretised in object space to generate image patches of both, the aerial oblique and the MM data. This step overcomes the large perspective differences between the two data sets dramatically and simplifies the registration problem. A thorough experimental section using difficult registration scenarios ascertains an inlier rate of about 80%.
Aerial oblique images bear a great potential for the adjustment task at hand. Not only can the number of correspondences between the aerial and MM data set be increased but it also allows for correspondences at different height levels which can potentially stabilise the geometry. Chapter five presents adjustment results for four different areas in Rotterdam showcasing the performance of the aerial nadir as well as the aerial oblique registration approach in various scenarios. The verification of the results using surveyed GCPs demonstrates that depending on the area and the setting low decimetre accuracy is well achievable.
In general, this research investigates the use of aerial images for the correction of MM imaging data. The development of novel techniques to deal with this non-standard registration problem is the focus of this research effort. The combination of image reprojection mechanics, guided matching strategies, and illumination-invariant similarity measures enables the identification of highly accurate correspondences between the aerial and the terrestrial data sets at hand. Since aerial images are widely available, frequently updated, and sensor systems are becoming more powerful, presented techniques demonstrate the feasibility to overcome geometric differences efficiently. Solving positioning issues in urban areas is not solely a research problem for terrestrial mapping but also for closely related fields and technologies, such as robotics, UAV photogrammetry, or autonomous driving. The utilisation of visual cues for the correction of platform trajectories is not only a viable but also a cost-efficient and accurate method, which may well experience a more widespread use in the future.
[1] K-dimensional tree
