Correlation of free-response and receiver-operating-characteristic area-under-the-curve estimates: Results from independently conducted FROCROC studies in mammography

Federica Zanca; Stephen L. Hillis; Filip Claus; Chantal Van Ongeval; Valerie Celis; Veerle Provoost; Hong Jun Yoon; Hilde Bosmans

doi:10.1118/1.4747262

Correlation of free-response and receiver-operating-characteristic area-under-the-curve estimates: Results from independently conducted FROCROC studies in mammography

Federica Zanca, Stephen L. Hillis, Filip Claus, Chantal Van Ongeval, Valerie Celis, Veerle Provoost, Hong Jun Yoon, Hilde Bosmans

Research output: Contribution to journal › Article › peer-review

10 Scopus citations

Abstract

Purpose: From independently conducted free-response receiver operating characteristic (FROC) and receiver operating characteristic (ROC) experiments, to study fixed-reader associations between three estimators: the area under the alternative FROC (AFROC) curve computed from FROC data, the area under the ROC curve computed from FROC highest rating data, and the area under the ROC curve computed from confidence-of-disease ratings. Methods: Two hundred mammograms, 100 of which were abnormal, were processed by two image-processing algorithms and interpreted by four radiologists under the FROC paradigm. From the FROC data, inferred-ROC data were derived, using the highest rating assumption. Eighteen months afterwards, the images were interpreted by the same radiologists under the conventional ROC paradigm; conventional-ROC data (in contrast to inferred-ROC data) were obtained. FROC and ROC (inferred, conventional) data were analyzed using the nonparametric area-under-the-curve (AUC), (AFROC and ROC curve, respectively). Pearson correlation was used to quantify the degree of association between the modality-specific AUC indices and standard errors were computed using the bootstrap-after-bootstrap method. The magnitude of the correlations was assessed by comparison with computed Obuchowski-Rockette fixed reader correlations. Results: Average Pearson correlations (with 95 confidence intervals in square brackets) were: Corr(FROC, inferred ROC) 0.760.64, 0.84 > Corr(inferred ROC, conventional ROC) 0.400.18, 0.58 > Corr (FROC, conventional ROC) 0.320.16, 0.46. Conclusions: Correlation between FROC and inferred-ROC data AUC estimates was high. Correlation between inferred- and conventional-ROC AUC was similar to the correlation between two modalities for a single reader using one estimation method, suggesting that the highest rating assumption might be questionable.

Original language	English
Pages (from-to)	5917-5929
Number of pages	13
Journal	Medical Physics
Volume	39
Issue number	10
DOIs	https://doi.org/10.1118/1.4747262
State	Published - Oct 2012
Externally published	Yes

Funding

F.Z. was supported in part by the Mevic project. Mevic is an IBBT-project in cooperation with the following companies and organizations: Barco, Hologic, Philips, University Hospital Leuven, University of Gent MEDISIP/IPI, Free University of Brussels, and ETRO. The IBBT is an independent multidisciplinary research institute founded by the Flemish government to stimulate ICT innovation. F.Z. wants to thank Jurgen Jacobs for the support with the use of the SARA v software. F.Z. would like to thank Professor Dev Chakraborty for his help and advice with the statistical analysis and the editing of the paper. S.L.H. was supported by the National Institutes of Health, R01-EB000863 and R01-EB013667. H.J.Y. was supported in part by grants from the Department of Health and Human Services, National Institutes of Health, Grant Nos. R01-EB005243 and R01-EB008688. 2

Funders	Funder number
National Institutes of Health	R01-EB000863
U.S. Department of Health and Human Services	R01-EB005243, R01-EB008688
National Institute of Biomedical Imaging and Bioengineering	R01EB013667

Keywords

AFROC
AUC estimates correlations
ROC

Access to Document

10.1118/1.4747262

Cite this

@article{e713379e9f264cad8caa160ce568d467,

title = "Correlation of free-response and receiver-operating-characteristic area-under-the-curve estimates: Results from independently conducted FROCROC studies in mammography",

abstract = "Purpose: From independently conducted free-response receiver operating characteristic (FROC) and receiver operating characteristic (ROC) experiments, to study fixed-reader associations between three estimators: the area under the alternative FROC (AFROC) curve computed from FROC data, the area under the ROC curve computed from FROC highest rating data, and the area under the ROC curve computed from confidence-of-disease ratings. Methods: Two hundred mammograms, 100 of which were abnormal, were processed by two image-processing algorithms and interpreted by four radiologists under the FROC paradigm. From the FROC data, inferred-ROC data were derived, using the highest rating assumption. Eighteen months afterwards, the images were interpreted by the same radiologists under the conventional ROC paradigm; conventional-ROC data (in contrast to inferred-ROC data) were obtained. FROC and ROC (inferred, conventional) data were analyzed using the nonparametric area-under-the-curve (AUC), (AFROC and ROC curve, respectively). Pearson correlation was used to quantify the degree of association between the modality-specific AUC indices and standard errors were computed using the bootstrap-after-bootstrap method. The magnitude of the correlations was assessed by comparison with computed Obuchowski-Rockette fixed reader correlations. Results: Average Pearson correlations (with 95 confidence intervals in square brackets) were: Corr(FROC, inferred ROC) 0.760.64, 0.84 > Corr(inferred ROC, conventional ROC) 0.400.18, 0.58 > Corr (FROC, conventional ROC) 0.320.16, 0.46. Conclusions: Correlation between FROC and inferred-ROC data AUC estimates was high. Correlation between inferred- and conventional-ROC AUC was similar to the correlation between two modalities for a single reader using one estimation method, suggesting that the highest rating assumption might be questionable.",

keywords = "AFROC, AUC estimates correlations, ROC",

author = "Federica Zanca and Hillis, {Stephen L.} and Filip Claus and {Van Ongeval}, Chantal and Valerie Celis and Veerle Provoost and Yoon, {Hong Jun} and Hilde Bosmans",

year = "2012",

month = oct,

doi = "10.1118/1.4747262",

language = "English",

volume = "39",

pages = "5917--5929",

journal = "Medical Physics",

issn = "0094-2405",

publisher = "wiley",

number = "10",

}

TY - JOUR

T1 - Correlation of free-response and receiver-operating-characteristic area-under-the-curve estimates

T2 - Results from independently conducted FROCROC studies in mammography

AU - Zanca, Federica

AU - Hillis, Stephen L.

AU - Claus, Filip

AU - Van Ongeval, Chantal

AU - Celis, Valerie

AU - Provoost, Veerle

AU - Yoon, Hong Jun

AU - Bosmans, Hilde

PY - 2012/10

Y1 - 2012/10

N2 - Purpose: From independently conducted free-response receiver operating characteristic (FROC) and receiver operating characteristic (ROC) experiments, to study fixed-reader associations between three estimators: the area under the alternative FROC (AFROC) curve computed from FROC data, the area under the ROC curve computed from FROC highest rating data, and the area under the ROC curve computed from confidence-of-disease ratings. Methods: Two hundred mammograms, 100 of which were abnormal, were processed by two image-processing algorithms and interpreted by four radiologists under the FROC paradigm. From the FROC data, inferred-ROC data were derived, using the highest rating assumption. Eighteen months afterwards, the images were interpreted by the same radiologists under the conventional ROC paradigm; conventional-ROC data (in contrast to inferred-ROC data) were obtained. FROC and ROC (inferred, conventional) data were analyzed using the nonparametric area-under-the-curve (AUC), (AFROC and ROC curve, respectively). Pearson correlation was used to quantify the degree of association between the modality-specific AUC indices and standard errors were computed using the bootstrap-after-bootstrap method. The magnitude of the correlations was assessed by comparison with computed Obuchowski-Rockette fixed reader correlations. Results: Average Pearson correlations (with 95 confidence intervals in square brackets) were: Corr(FROC, inferred ROC) 0.760.64, 0.84 > Corr(inferred ROC, conventional ROC) 0.400.18, 0.58 > Corr (FROC, conventional ROC) 0.320.16, 0.46. Conclusions: Correlation between FROC and inferred-ROC data AUC estimates was high. Correlation between inferred- and conventional-ROC AUC was similar to the correlation between two modalities for a single reader using one estimation method, suggesting that the highest rating assumption might be questionable.

AB - Purpose: From independently conducted free-response receiver operating characteristic (FROC) and receiver operating characteristic (ROC) experiments, to study fixed-reader associations between three estimators: the area under the alternative FROC (AFROC) curve computed from FROC data, the area under the ROC curve computed from FROC highest rating data, and the area under the ROC curve computed from confidence-of-disease ratings. Methods: Two hundred mammograms, 100 of which were abnormal, were processed by two image-processing algorithms and interpreted by four radiologists under the FROC paradigm. From the FROC data, inferred-ROC data were derived, using the highest rating assumption. Eighteen months afterwards, the images were interpreted by the same radiologists under the conventional ROC paradigm; conventional-ROC data (in contrast to inferred-ROC data) were obtained. FROC and ROC (inferred, conventional) data were analyzed using the nonparametric area-under-the-curve (AUC), (AFROC and ROC curve, respectively). Pearson correlation was used to quantify the degree of association between the modality-specific AUC indices and standard errors were computed using the bootstrap-after-bootstrap method. The magnitude of the correlations was assessed by comparison with computed Obuchowski-Rockette fixed reader correlations. Results: Average Pearson correlations (with 95 confidence intervals in square brackets) were: Corr(FROC, inferred ROC) 0.760.64, 0.84 > Corr(inferred ROC, conventional ROC) 0.400.18, 0.58 > Corr (FROC, conventional ROC) 0.320.16, 0.46. Conclusions: Correlation between FROC and inferred-ROC data AUC estimates was high. Correlation between inferred- and conventional-ROC AUC was similar to the correlation between two modalities for a single reader using one estimation method, suggesting that the highest rating assumption might be questionable.

KW - AFROC

KW - AUC estimates correlations

KW - ROC

UR - http://www.scopus.com/inward/record.url?scp=84867288387&partnerID=8YFLogxK

U2 - 10.1118/1.4747262

DO - 10.1118/1.4747262

M3 - Article

C2 - 23039631

AN - SCOPUS:84867288387

SN - 0094-2405

VL - 39

SP - 5917

EP - 5929

JO - Medical Physics

JF - Medical Physics

IS - 10

ER -

Correlation of free-response and receiver-operating-characteristic area-under-the-curve estimates: Results from independently conducted FROCROC studies in mammography

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this