DEBRIS, RUBBLE PILES AND FAÇADE DAMAGE DETECTION USING MULTI-RESOLUTION optical REMOTE SENSING IMAGERY
Diogo Duarte is a PhD student in the Dpartment of Earth Systems Analysis (ESA). His supervisors are prof.dr.ir. M.G. Vosselman and prof.dr. N. Kerle from the Faculty of Geo-Information Science and Earth Observation (ITC).
Knowledge of the location of damaged buildings is of utmost importance for both the response and recovery phases of the disaster management cycle. To this regard, remote sensing images have been continuously used over the last 20 years as the main data source in approaches to detect building damages. Partially and totally collapsed buildings are the structures which might contain entrapped victims; hence many studies focus on the mapping of debris and rubble piles. Nonetheless such assumption might leave out damage evidences such as spalling or cracks, especially in the façades. When comparing with the mapping of rubble piles and debris, the façade damage detection is an understudied topic. The objective of the research reported in this thesis was focused on the mapping of both debris/ruble piles and façade damages from remote sensing images.
The mapping of partially and totally collapsed buildings is often constrained by the used system (platform and sensor). There is a growing amount of imagery being collected (e.g. by the International Charter and Emergency Management Service) using different sensors, platforms and resolution, where their optimal use and integration would represent an opportunity to positively impact the detection of building damages. However, this multitude of systems does not imply the availability of large datasets sufficient to train recent and more complex algorithms such as convolutional neural networks (CNN). Hence, one of the goals of this thesis is to fuse satellite and aerial (manned and unmanned) image samples in a unique classification network to assess the building damage detection in each of the considered resolution levels.
While there are several contributions regarding the mapping of debris and rubble piles, this is not the case when focusing on the specific case of façade damages. Nonetheless, façade image data are already being collected by both aerial manned and unmanned vehicles. Regarding the use of UAV, only a few approaches focused on the specific issue of façade damage detection. These are often not made operational and require computationally expensive procedures which limit their utility to stakeholders, who need fast and reliable façade damage information. One of the objectives of this thesis is to improve the efficiency of such façade damage detection procedures. On the other hand, aerial manned platforms have a wider coverage whilst capturing data at a lower resolution. In particular, the use of imagery coming from aerial (manned) oblique surveys has substantially increased in the last decade, leading to periodic aerial surveys over entire cities in many countries. Such data could be therefore exploited for multi-temporal image classification of façade damages over a given city/region. This was the main focus of the third goal of this thesis, the detection of façade damages mainly focusing on the use of multi-temporal aerial oblique imagery to infer on the damage state of a given façade.
Related with the overall objective of mapping rubble piles, debris and façade damages, three distinct objectives, with their own set of experiments are investigated in this thesis:
1. Mapping of partially and totally collapsed buildings using multi-resolution remote sensing images (Chapters 2 and 3).
A preliminary study regarding the use of multi-resolution imagery focused on the specific case of the satellite image classification of building damages. Features were extracted from satellite and aerial (manned and unmanned) imagery and fed to a supervised classifier to detect rubble piles and debris in satellite images. The approaches considering image samples coming from other resolutions outperformed the traditional approach by nearly 4%, where the traditional approaches used only satellite image samples during training. Picking up on these results, the approach was extended to the other resolutions, referring to aerial manned and unmanned. Using the multi-resolution approach for the image classification of debris and rubble piles, improved the results in the case of aerial unmanned (by ~5 %) and performed similarly to traditional approaches when using aerial manned platforms. The best performing multi-resolution approach merged the features coming from the three different sets of images, and also considered feature information from the intermediate layers of each of the levels of resolution. The approach was also tested for geographical transferability where the differences between the traditional and multi-resolution approaches were maintained.
2. Efficient detection of earthquake induced façade damages from UAV images (Chapter 4):
An approach to perform a more efficient detection of façade damages from UAV images was developed. It aimed at reducing the time between the deployment of the UAV and per façade damage results, in order to be of use to first responders Such efficiency was achieved by directing all damage classification computations to the specific image regions containing the façades. This was achieved by acquiring nadir images in a first flight, which allowed to detect the buildings and consequently define the façades. This 3D façade information was then used to identify the façades in oblique images acquired in a second flight from which façade damages could be assessed. The buildings were identified by segmenting the building roofs from the sparse point cloud directly, avoiding the computationally expensive dense image matching algorithm. The acquired data were georeferenced using the on-board information. The second flight was performed only on façades of interest, where all the damage detection procedures were only applied to these same façades. Although this method is more efficient, the detection of façade damages used a model trained only on rubble piles and debris, delivering a high rate of false positives and leaving out smaller cues of damage such as spalling or cracks. This method is only achieving ~80% accuracy.
3. Multi-temporal façade damage detection (Chapters 5 and 6):
The last objective focused on the use of multi-temporal aerial oblique datasets to assess a given city/region for façade damages. The first step in the detection of façade damages was the extraction of oblique image patches depicting the façade. To achieve this, the pre-event point cloud was generated through dense image matching, where the rest of the approach followed a similar façade extraction procedure as indicated in 2). Preliminary results on the multi-temporal façade damage detection were obtained by comparing rectified façade image patches, between and within epochs, using a simple cross correlation coefficient. This multi-temporal study was further investigated by integrating it in a supervised classification approach using CNN. This approach focused on two main issues: (i) the optimal fusion the multi-temporal data and (ii) the use of high overlapping aerial images to extract the same façade from different views and embedding them in the multi-temporal approach. The results demonstrated the benefits given by façade damage detection approaches using multi-temporal datasets. Moreover, the results show that considering several views per façade within a CNN approach improves the image classification of façade damages. The multi-temporal approach outperformed the mono-temporal ones by 20% in f1-score, where the best multi-temporal approach achieved an f1-score of 82%. Given the limited number of samples and the relatively low resolution, smaller damage evidences such as small cracks and/or small areas of spalling could not be detected.
The research reported in this thesis was part of the EU (7th Framework Programme) funded INACHUS (Technological and Methodological Solutions for Integrated Wide Area Situation Awareness and Survivor Localization to Support Search and Rescue Teams) project (www.inachus.eu). This project aimed at a time reduction of the response phase performed by FR, namely in the identification of entrapped victims after a disaster. The work reported in this thesis focused on the use of aerial imagery to localize damaged buildings over a given region/building block.