Evaluating classifiers: Relation between area under the receiver operator characteristic curve and overall accuracy

In this study, we investigated the relation between two popular classifier performance measures: area under the receiver operator characteristic curve and overall accuracy. We also evaluated the impact of class imbalance and number of examples in test set on this relation. We perform a set of experiments in which we train multiple neural networks and test them in various, well controlled conditions. The experimental results show that given a large and balanced test set, increase in one performance measure is a very good indicator of increase in the other measure. Furthermore increasing the total number of examples, while keeping the positive class prevalence constant generally increases the correlation between the two measures. Our results also indicate that increasing the extent of class imbalance in the test set has a detrimental effect on this correlation.