Investigation of methods for analyzing location specific observer performance data

We examined the statistical powers of three methods for analyzing FROC mark-rating data, namely ROC, JAFROC and IDCA. Two classes of observers were simulated: a designer-level CAD algorithm and a human observer. A search-model based simulator was used with the average numbers of false positives per image ranging from 0.21 for the human observer to 10 for CAD. Model parameters were chosen to yield 80% and 85% areas under the predicted ROC curves for both classes of observers and inter-image and inter-modality correlations of 0.1, 0.5 and 0.9 were investigated. The area under the FROC curve up to abscissa a (ranging from 0.18 to 6.7) was used as the IDCA figure-of-merit; the other methods used their well-known figures of merit. For IDCA power increased with a so it should be chosen as large as possible consistent with the need for overlap of the two FROC curves in the x-direction. For CAD the IDCA method yielded the highest statistical power. Surprisingly, JAFROC yielded the highest statistical power for human observers, even greater than IDCA which, unlike JAFROC, uses all the marks. The largest difference occurred for conservative reporting styles and high data correlation: e.g., 0.3453 for JAFROC vs. 0.2672 for IDCA. One reason is that unlike IDCA, the JAFROC figure of merit is sensitive to unmarked normal images and unmarked lesions. In all cases the ROC method yielded the least statistical power and entailed a substantial statistical power penalty (e.g., 24% for ROC vs. 41% for JAFROC). For human observers JAFROC should be used and for designer-level CAD data IDCA should be used and use of the ROC method for localization studies is discouraged.