A Routine and Post-disaster Road Corridor Monitoring Framework for the Increased Resilience of Road Infrastructures
Sofia Tilon is a PhD student in the department of Applied Earth Sciences. Promotors are prof.dr. N. Kerle, prof.dr.ir. M.G. Vosselman and prof.dr.ing. F.C. Nex from the faculty Geo-information and Earth Observation (ITC).
Resilient road infrastructures are vital to a society's welfare and economy; however, monitoring and maintaining them is becoming increasingly complex due to climate change, ageing infrastructure assets and increased traffic loads. Increasing maintenance expenditure is required to refurbish assets, of which the majority are already reaching the end of their predicted service life, and to make them resilient against (future) stressors that lead to accelerated degradation and damage. Monitoring is crucial in assessing the state and performance level of infrastructures. However, most Western countries have deferred essential maintenance and monitoring. Catastrophic events in recent years have highlighted the state of today's crumbling infrastructure and made it evident that most countries face the critical challenge of upkeeping these vital “lifelines”. There is a need for improved, automated, accurate and synoptic monitoring technologies to support road operators in keeping road infrastructures safe and functional in routine and emergency scenarios. Modern inspection technology increasingly supports human-based inspections and is extremely valuable in alleviating the limitations that exist in operational or traditional monitoring methods. There are various state-of-the-art technologies and perspectives from which infrastructure monitoring can be regarded. This dissertation considers it in a non-destructive and remote sensing manner while simultaneously meeting the demand of road operators to achieve monitoring in a real-time, multi-objective and automatic manner. To this end, research was conducted to investigate and propose novel monitoring technology to aid road infrastructure monitoring in routine and emergency scenarios using earth observation platforms and Artificial Intelligence (AI).
The following objectives were investigated:
1. To determine the applicability of anomaly-detecting generative adversarial networks (ADGANs) for routine and post-disaster damage assessments. (Chapters 2 and 3)
Machine learning for road infrastructure damage detection suffers from insufficient training data containing examples of damages that show a broad range of visual appearances and damage types. This dissertation investigated the usability of ADGANs, a state-of-the-art unsupervised approach to detect damage from imagery from Unmanned Aerial Vehicles (UAVs), satellites and ground vehicles. The two selected and tested ADGANs were GANomaly and Skip-GANomaly. The primary approach towards testing these ADGANs in Chapter 2 and Chapter 3 was to evaluate their ability to generate realistic imagery of undamaged infrastructure scenes, to assess their ability to differentiate between damaged and undamaged images and to investigate their performance when the data were pre-processed to eliminate well-known problem categories (shadows, vegetation, or road markings). In Chapter 3, a particular focus was placed on the transferability of the models trained on satellite data to different geographical locations or disaster events. In addition, image differencing was used to achieve damage localisation.
The conclusion was that ADGAN-based building damage detection from satellite imagery was successful when vegetation and shadows were excluded from the training process. Furthermore, damages could be better detected when the damage signal was spread at large scales, across a bigger area, and not limited to the building footprint. For example, damages induced by flood and wildfire events were better detected than damages from other disasters. Degradation could not be detected using ADGANs and high-resolution imagery from ground vehicles. It is hypothesised that the visual contrast between the degradation signal and the surrounding areas is too low for the ADGAN to delineate between degradation and regular asphalt patches effectively. ADGAN-based building damage detection from UAVs proved successful on the condition that shadows and vegetation were excluded from the training phase. The conclusion is drawn that the large-scale damage signal, i.e., the damage signal expressed across an area, was expressive enough to effectively differentiate between “normal” and “abnormal” areas.
2. To develop a framework for real-time infrastructure monitoring using (hybrid) UAVs. (Chapter 4)
The UAV domain has undergone several advances in design and utilisation, affecting infrastructure monitoring differently. UAV designs traditionally come in two forms: vertical take-off and landing (VTOL) and fixed-wing UAVs. A new type of UAV, the fixed-wing VTOL, also called “hybrid UAV” combines the characteristics of both traditional designs. It could achieve lift-off in confined spaces while also being able to fly for extended times with high payloads. This made hybrid UAVs especially useful for infrastructure monitoring.
The DeltaQuad fixed-wing VTOL was enhanced with edge computation capabilities, and a UAV-monitoring system was designed to carry out multiple objectives: 3D reconstruction and deep learning. The framework made optimal use of the hardware within the designed framework by allocating tasks to be executed at the most appropriate location: on-the-edge or remote workstation. Meanwhile, attention was given to practical factors, such as the end user's requirements or conditions at the scene, such as poor connectivity. Therefore, only the end-user relevant information, obtained using the onboard deep learning suite, was streamed to the ground control station or the remote station. Finally, the framework was tested and validated in a real-highway scenario. The conclusion was drawn that the designed system worked, while areas for improvement were identified.
3. To provide road operators with qualitative high-level information products to detect anomalous infrastructure scenarios from a UAV platform. (Chapter 5)
The final topic of this dissertation aimed at bridging the semantic gap observed in most monitoring methodologies that failed to translate AI-derived information products into qualitative and actionable knowledge items. At the same time, it aimed at satisfying the multitude of objectives road operators have that often compete in terms of resources or level of importance. Multi-task convolutional neural networks (CNNs) for joint object detection and scene segmentation and rule sets for semantic reasoning were applied to achieve that. The main benefit of using a multi-task framework was that multiple information items could be extracted in a single forward pass reducing the computational overhead and making it especially fit for deployment on edge devices.
The conclusion was that multi-task learning could obtain semantic scene segmentation and vehicle object detection in semi-real time on an edge device. The performance obtained for each individual task was lower than single-task CNNs, however, the main benefit was that multiple information objects could be extracted simultaneously. In addition, the semantic reasoner could successfully detect larger deviations, debris or anomalous vehicle speeds from a real-world infrastructure scene.
This dissertational research was carried out with the support of and in collaboration with the PANOPTIS project, ”Development of a Decision Support Framework for Increasing the Resilience of Transportation Infrastructure Based on Combined Use of Terrestrial and Airborne Sensors and Advanced Modelling Tools”, funded under the European Union’s Horizon 2020 research and innovation programme.
