Satellite SAR-optical image fusion for cloud-free image generation
Chenxi Duan is a PhD student in the Department of Earth Observation Science. Promotors are dr. M. Belgiu and prof.dr. A. Stein from the Faculty Geo-Information Science and Earth Observation.
This dissertation tackles the critical and persistent problem of missing or cloud-obscured optical satellite images. Clouds significantly impede the acquisition of complete and timely images for many Earth observation applications. Remote sensing methods on incomplete images often face challenges, especially for applications demanding comprehensive coverage like change detection or environmental monitoring. Unfortunately, obtaining consistently cloud-free satellite images is often impossible for specific periods or regions. This thesis explores how to enhance the effectiveness and reliability of these images. Effectiveness is the ability to reconstruct cloud-contaminated areas and to enable their application in real-world remote sensing tasks. Reliability refers to the trustworthiness of the reconstructed information, meaning the generated images well represent the Earth’s surface and can be used for subsequent analysis. To achieve this, we propose multi-source data fusion methods, exploring these enhancements in practical applications such as crop mapping and post-flood optical image reconstruction. The thesis addresses four objectives.
First, most cloud removal methods fusing SAR images with optical images struggle to balance accuracy with real-world applicability and computational efficiency, especially when dealing with large study areas and heavy computations. This research part addresses these limitations by developing an innovative, computationally efficient, and accurate SAR-optical fusion network. For this research, we propose the Feature Enhancement Network (FENet) that learns a residual map, allowing it to preserve already clear pixels while precisely reconstructing obscured regions. It integrates a sophisticated Linear Attention Mechanism (LAM) to effectively capture global contextual information, which boosts reconstruction accuracy. Experiments, detailed in Chapter 2, demonstrate that Feature Enhancement Network (FENet) achieves a high accuracy, showing superior visual consistency and detail compared to existing methods. FENet is optimized for computational efficiency, resulting in faster processing and reduced training times, making it highly suitable for practical, large-scale deployment and reliable delivery of cloud-free data with minimal spectral distortions.
Second, efficient processing of large volumes of cloud-contaminated satellite images is critical in real-world applications. To minimize both time and computational resource consumption, the second part of this research focuses on developing optimized network architectures for SAR-optical image fusion that explicitly prioritize efficiency. The Feature Pyramid Network (FPNet) is proposed to meet these practical demands. FPNet’s core lies in combining its unique feature pyramid structure with its specific fusion design. Being engineered for highly efficient multi-scale feature extraction and data integration, it substantially reduces the computational workload. This fusion design allows for the effective combination of information from SAR and optical images from different scales. Experiments, detailed in Chapter 3, confirm FPNet’s remarkable efficiency, achieving inference speeds of up to 96 FPS and drastically cutting down training times.
Third, methods for cloud removal and image reconstruction have advanced, but their direct impact on the accuracy of subsequent downstream applications, such as crop type mapping, often remains underexplored. This research part specifically investigates the use of reconstructed cloud-free images for improving the reliability and accuracy of agricultural land classification. The study assesses how the quality of images obtained through SAR-optical data fusion-based cloud removal and reconstruction methods translates into enhanced crop mapping performance. Our findings, presented in Chapter 4, demonstrate that the improved data quality from these advanced reconstruction techniques modestly contributes to more accurate and robust crop type identification.
Fourth, the task of recovering complete optical satellite images is frequently required due to the presence of clouds. Existing cloud removal methods, including those previously discussed, often fall short in effectively achieving full-scene image generation under such challenging conditions. To address this, the Cloud-Resilient Generation Network (CRGenNet) is proposed as a novel method that uses two temporal and two modal images (both SAR and optical) at two acquisition times to fuse information from SAR and optical domains. CRGenNet incorporates two key components: DownUpBlock for early and effective fusion of SAR and optical information, specifically designed to handle clouds present in the auxiliary optical image, and FusionAttention to improve multi-temporal feature integration. The method enables the complete recovery of an entire missing image. Our experiments, detailed in Chapter 5, consistently demonstrate high performance across diverse land cover types and challenging environmental conditions, highlighting CRGenNet’s broad applicability and strong robustness.
To summarize, this thesis presents novel methods based upon deep learning-based significantly enhance quality of optical satellite images by effectively addressing the challenges of cloud contamination. These methods are important for scientific and operational endeavors aiming to obtain complete and reliable Earth observation images, especially if consistently clear optical images are scarce.
