A novel technique for assessing the case-specific reliability of decisions made by CAD tools

We present a novel technique that provides a case-specific confidence measure for artificial neural network (ANN) based computer-assisted diagnosis (CAD) decisions. It relies on the analysis of the feature space neighborhood for each query case and dynamically creates a validation set that allows estimation of a local accuracy of the decisions made by the network. Then a case-specific reliability measure is assigned to each system's response, which can be used to stratify network's predictions according to the acceptable validation error value. The study was performed using a database containing 1,337 mammographic regions of interest (ROIs) with biopsyproven diagnosis (681 with masses, 656 with normal parenchyma). Two types of neural networks (1) a feed forward network with error back propagation (BPNN) and (2) a generalized regression neural network with RBF nodes (GRNN) were developed to detect masses based on 8 morphological features automatically extracted from each ROI. The performance of the networks was evaluated with Receiver Operating Characteristics (ROC) analysis. The study shows that as the threshold on the acceptable validation error declines, the technique rejects more CAD decisions as not reliable enough. However, the ROC performance for the reliable results steadily improves (from A _z = 0.88 to A _z = 0.98 for BPNN, from A _z = 0.86 to A _z = 0.97 for GRNN). The proposed technique provides a stratification strategy for predictions made by CAD tools and can be applied to any type of decision algorithms.