Efficient Data Source Relevance Quantification for Multi-Source Neural Networks

In deep learning-based data fusion, accurately quantifying the relevance of individual data sources offers enhanced analytical capabilities and insights into source-wise information. However, current methodologies are computationally expensive, requiring multiple forward passes or separate backpropagation for each output. In this paper, we present Relevance Forward Propagation (RFP). This approach efficiently computes data source relevance values for all network outputs on the fly within a single forward pass by utilizing the aggregation of source-wise relevance values. We mathematically prove and experimentally validate the equivalence of the resulting source-wise relevance values to those computed with the well-established backpropagation-based Layer-Wise Relevance Propagation. We validate the effectiveness and efficiency of RFP against several existing approaches. Using a data fusion MNIST, we explore factors affecting data source relevance, such as noise and disparities in data source complexities. We extend these insights to practical domains by addressing the fusion of satellite data from Synthetic Aperture Radar (SAR) and optical satellites. Our method demonstrates adaptability within complex settings in scenarios where clouds affect optical data. Our proposed approach efficiently quantifies the relevance values of individual data sources within the prediction step of deep learning-based fusion, extending to real-world complexities and challenges encountered in satellite data fusion. Code and examples are available at github.com/JakobCode/RFP.