Privacy-Preserving Knowledge Transfer with Bootstrap Aggregation of Teacher Ensembles

Hong Jun Yoon; Hilda B. Klasky; Eric B. Durbin; Xiao Cheng Wu; Antoinette Stroup; Jennifer Doherty; Linda Coyle; Lynne Penberthy; Christopher Stanley; J. Blair Christian; Georgia D. Tourassi

doi:10.1007/978-3-030-71055-2_9

Privacy-Preserving Knowledge Transfer with Bootstrap Aggregation of Teacher Ensembles

Hong Jun Yoon, Hilda B. Klasky, Eric B. Durbin, Xiao Cheng Wu, Antoinette Stroup, Jennifer Doherty, Linda Coyle, Lynne Penberthy, Christopher Stanley, J. Blair Christian, Georgia D. Tourassi

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

There is a need to transfer knowledge among institutions and organizations to save effort in annotation and labeling or in enhancing task performance. However, knowledge transfer is difficult because of restrictions that are in place to ensure data security and privacy. Institutions are not allowed to exchange data or perform any activity that may expose personal information. With the leverage of a differential privacy algorithm in a high-performance computing environment, we propose a new training protocol, Bootstrap Aggregation of Teacher Ensembles (BATE), which is applicable to various types of machine learning models. The BATE algorithm is based on and provides enhancements to the PATE algorithm, maintaining competitive task performance scores on complex datasets with underrepresented class labels. We conducted a proof-of-the-concept study of the information extraction from cancer pathology report data from four cancer registries and performed comparisons between four scenarios: no collaboration, no privacy-preserving collaboration, the PATE algorithm, and the proposed BATE algorithm. The results showed that the BATE algorithm maintained competitive macro-averaged F1 scores, demonstrating that the suggested algorithm is an effective yet privacy-preserving method for machine learning and deep learning solutions.

Original language	English
Title of host publication	Heterogeneous Data Management, Polystores, and Analytics for Healthcare - VLDB Workshops, Poly 2020 and DMAH 2020, Revised Selected Papers
Editors	Vijay Gadepally, Timothy Mattson, Michael Stonebraker, Tim Kraska, Fusheng Wang, Gang Luo, Jun Kong, Alevtina Dubovitskaya
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	87-99
Number of pages	13
ISBN (Print)	9783030710545
DOIs	https://doi.org/10.1007/978-3-030-71055-2_9
State	Published - 2021
Event	VLDB workshops: International Workshop on Polystore Systems for Heterogeneous Data in Multiple Databases with Privacy and Security Assurances, Poly 2020, and 6th International Workshop on Data Management and Analytics for Medicine and Healthcare, DMAH 2020 - Virtual, Online Duration: Aug 31 2020 → Sep 4 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12633 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	VLDB workshops: International Workshop on Polystore Systems for Heterogeneous Data in Multiple Databases with Privacy and Security Assurances, Poly 2020, and 6th International Workshop on Data Management and Analytics for Medicine and Healthcare, DMAH 2020
City	Virtual, Online
Period	08/31/20 → 09/4/20

Funding

This research used resources of the Oak Ridge Leadership Computing Facility at ORNL, which is supported by the DOE Office of Science under Contract No. DE-AC05-00OR22725. The study was supported by the Laboratory Directed Research and Development (LDRD) program of Oak Ridge National Laboratory, under LDRD project No. 9831. LTR data were collected using funding from NCI and the Surveillance, Epidemiology and End Results (SEER) Program (HHSN261201800007I), the CDC’s National Program of Cancer Registries (NPCR) (NU58DP006332-02-00) as well as the State of Louisiana. Acknowledgement. This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the US Department of Energy (DOE) Office of Science and the National Nuclear Security Administration. This work has been supported in part by the Joint Design of Advanced Computing Solutions for Cancer (JDACS4C) program established by DOE and the National Cancer Institute of the National Institutes of Health. This work was performed under the auspices of DOE by Argonne National Laboratory under Contract DE-AC02-06-CH11357, Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344, Los Alamos National Laboratory under Contract DE-AC5206NA25396, and ORNL under Contract DE-AC05-00OR22725. NJSCR data were collected using funding from NCI and the Surveillance, Epidemiology and End Results (SEER) Program (HHSN261201300021I), the CDC’s National Program of Cancer Registries (NPCR) (NU58DP006279-02-00) as well as the State of New Jersey and the Rutgers Cancer Institute of New Jersey. KCR data were collected with funding from NCI Surveillance, Epidemiology and End Results (SEER) Program (HHSN261201800013I), the CDC National Program of Cancer Registries (NPCR) (U58DP00003907) and the Commonwealth of Kentucky. The Utah Cancer Registry is funded by the National Cancer Institute’s SEER Program, Contract No. HHSN261201800016I, and the US Centers for Disease Control and Prevention’s National Program of Cancer Registries, Cooperative Agreement No. NU58DP0063200, with additional support from the University of Utah and Huntsman Cancer Foundation.

Keywords

Bootstrap aggregation
Data privacy
Differential privacy
Information extraction
Natural language processing
Privacy-preserving machine learning

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1007/978-3-030-71055-2_9

Cite this

Yoon, H. J., Klasky, H. B., Durbin, E. B., Wu, X. C., Stroup, A., Doherty, J., Coyle, L., Penberthy, L., Stanley, C., Christian, J. B., & Tourassi, G. D. (2021). Privacy-Preserving Knowledge Transfer with Bootstrap Aggregation of Teacher Ensembles. In V. Gadepally, T. Mattson, M. Stonebraker, T. Kraska, F. Wang, G. Luo, J. Kong, & A. Dubovitskaya (Eds.), Heterogeneous Data Management, Polystores, and Analytics for Healthcare - VLDB Workshops, Poly 2020 and DMAH 2020, Revised Selected Papers (pp. 87-99). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12633 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-71055-2_9

Yoon, Hong Jun ; Klasky, Hilda B. ; Durbin, Eric B. et al. / Privacy-Preserving Knowledge Transfer with Bootstrap Aggregation of Teacher Ensembles. Heterogeneous Data Management, Polystores, and Analytics for Healthcare - VLDB Workshops, Poly 2020 and DMAH 2020, Revised Selected Papers. editor / Vijay Gadepally ; Timothy Mattson ; Michael Stonebraker ; Tim Kraska ; Fusheng Wang ; Gang Luo ; Jun Kong ; Alevtina Dubovitskaya. Springer Science and Business Media Deutschland GmbH, 2021. pp. 87-99 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{7c7113c2dff244e69e264fddb5fa28e2,

title = "Privacy-Preserving Knowledge Transfer with Bootstrap Aggregation of Teacher Ensembles",

abstract = "There is a need to transfer knowledge among institutions and organizations to save effort in annotation and labeling or in enhancing task performance. However, knowledge transfer is difficult because of restrictions that are in place to ensure data security and privacy. Institutions are not allowed to exchange data or perform any activity that may expose personal information. With the leverage of a differential privacy algorithm in a high-performance computing environment, we propose a new training protocol, Bootstrap Aggregation of Teacher Ensembles (BATE), which is applicable to various types of machine learning models. The BATE algorithm is based on and provides enhancements to the PATE algorithm, maintaining competitive task performance scores on complex datasets with underrepresented class labels. We conducted a proof-of-the-concept study of the information extraction from cancer pathology report data from four cancer registries and performed comparisons between four scenarios: no collaboration, no privacy-preserving collaboration, the PATE algorithm, and the proposed BATE algorithm. The results showed that the BATE algorithm maintained competitive macro-averaged F1 scores, demonstrating that the suggested algorithm is an effective yet privacy-preserving method for machine learning and deep learning solutions.",

keywords = "Bootstrap aggregation, Data privacy, Differential privacy, Information extraction, Natural language processing, Privacy-preserving machine learning",

author = "Yoon, {Hong Jun} and Klasky, {Hilda B.} and Durbin, {Eric B.} and Wu, {Xiao Cheng} and Antoinette Stroup and Jennifer Doherty and Linda Coyle and Lynne Penberthy and Christopher Stanley and Christian, {J. Blair} and Tourassi, {Georgia D.}",

note = "Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.; VLDB workshops: International Workshop on Polystore Systems for Heterogeneous Data in Multiple Databases with Privacy and Security Assurances, Poly 2020, and 6th International Workshop on Data Management and Analytics for Medicine and Healthcare, DMAH 2020 ; Conference date: 31-08-2020 Through 04-09-2020",

year = "2021",

doi = "10.1007/978-3-030-71055-2_9",

language = "English",

isbn = "9783030710545",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "87--99",

editor = "Vijay Gadepally and Timothy Mattson and Michael Stonebraker and Tim Kraska and Fusheng Wang and Gang Luo and Jun Kong and Alevtina Dubovitskaya",

booktitle = "Heterogeneous Data Management, Polystores, and Analytics for Healthcare - VLDB Workshops, Poly 2020 and DMAH 2020, Revised Selected Papers",

}

Yoon, HJ , Klasky, HB, Durbin, EB, Wu, XC, Stroup, A, Doherty, J, Coyle, L, Penberthy, L, Stanley, C, Christian, JB & Tourassi, GD 2021, Privacy-Preserving Knowledge Transfer with Bootstrap Aggregation of Teacher Ensembles. in V Gadepally, T Mattson, M Stonebraker, T Kraska, F Wang, G Luo, J Kong & A Dubovitskaya (eds), Heterogeneous Data Management, Polystores, and Analytics for Healthcare - VLDB Workshops, Poly 2020 and DMAH 2020, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12633 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 87-99, VLDB workshops: International Workshop on Polystore Systems for Heterogeneous Data in Multiple Databases with Privacy and Security Assurances, Poly 2020, and 6th International Workshop on Data Management and Analytics for Medicine and Healthcare, DMAH 2020, Virtual, Online, 08/31/20. https://doi.org/10.1007/978-3-030-71055-2_9

Privacy-Preserving Knowledge Transfer with Bootstrap Aggregation of Teacher Ensembles. / Yoon, Hong Jun ; Klasky, Hilda B.; Durbin, Eric B. et al.
Heterogeneous Data Management, Polystores, and Analytics for Healthcare - VLDB Workshops, Poly 2020 and DMAH 2020, Revised Selected Papers. ed. / Vijay Gadepally; Timothy Mattson; Michael Stonebraker; Tim Kraska; Fusheng Wang; Gang Luo; Jun Kong; Alevtina Dubovitskaya. Springer Science and Business Media Deutschland GmbH, 2021. p. 87-99 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12633 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Privacy-Preserving Knowledge Transfer with Bootstrap Aggregation of Teacher Ensembles

AU - Yoon, Hong Jun

AU - Klasky, Hilda B.

AU - Durbin, Eric B.

AU - Wu, Xiao Cheng

AU - Stroup, Antoinette

AU - Doherty, Jennifer

AU - Coyle, Linda

AU - Penberthy, Lynne

AU - Stanley, Christopher

AU - Christian, J. Blair

AU - Tourassi, Georgia D.

PY - 2021

Y1 - 2021

N2 - There is a need to transfer knowledge among institutions and organizations to save effort in annotation and labeling or in enhancing task performance. However, knowledge transfer is difficult because of restrictions that are in place to ensure data security and privacy. Institutions are not allowed to exchange data or perform any activity that may expose personal information. With the leverage of a differential privacy algorithm in a high-performance computing environment, we propose a new training protocol, Bootstrap Aggregation of Teacher Ensembles (BATE), which is applicable to various types of machine learning models. The BATE algorithm is based on and provides enhancements to the PATE algorithm, maintaining competitive task performance scores on complex datasets with underrepresented class labels. We conducted a proof-of-the-concept study of the information extraction from cancer pathology report data from four cancer registries and performed comparisons between four scenarios: no collaboration, no privacy-preserving collaboration, the PATE algorithm, and the proposed BATE algorithm. The results showed that the BATE algorithm maintained competitive macro-averaged F1 scores, demonstrating that the suggested algorithm is an effective yet privacy-preserving method for machine learning and deep learning solutions.

AB - There is a need to transfer knowledge among institutions and organizations to save effort in annotation and labeling or in enhancing task performance. However, knowledge transfer is difficult because of restrictions that are in place to ensure data security and privacy. Institutions are not allowed to exchange data or perform any activity that may expose personal information. With the leverage of a differential privacy algorithm in a high-performance computing environment, we propose a new training protocol, Bootstrap Aggregation of Teacher Ensembles (BATE), which is applicable to various types of machine learning models. The BATE algorithm is based on and provides enhancements to the PATE algorithm, maintaining competitive task performance scores on complex datasets with underrepresented class labels. We conducted a proof-of-the-concept study of the information extraction from cancer pathology report data from four cancer registries and performed comparisons between four scenarios: no collaboration, no privacy-preserving collaboration, the PATE algorithm, and the proposed BATE algorithm. The results showed that the BATE algorithm maintained competitive macro-averaged F1 scores, demonstrating that the suggested algorithm is an effective yet privacy-preserving method for machine learning and deep learning solutions.

KW - Bootstrap aggregation

KW - Data privacy

KW - Differential privacy

KW - Information extraction

KW - Natural language processing

KW - Privacy-preserving machine learning

UR - http://www.scopus.com/inward/record.url?scp=85103597093&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-71055-2_9

DO - 10.1007/978-3-030-71055-2_9

M3 - Conference contribution

AN - SCOPUS:85103597093

SN - 9783030710545

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 87

EP - 99

BT - Heterogeneous Data Management, Polystores, and Analytics for Healthcare - VLDB Workshops, Poly 2020 and DMAH 2020, Revised Selected Papers

A2 - Gadepally, Vijay

A2 - Mattson, Timothy

A2 - Stonebraker, Michael

A2 - Kraska, Tim

A2 - Wang, Fusheng

A2 - Luo, Gang

A2 - Kong, Jun

A2 - Dubovitskaya, Alevtina

PB - Springer Science and Business Media Deutschland GmbH

T2 - VLDB workshops: International Workshop on Polystore Systems for Heterogeneous Data in Multiple Databases with Privacy and Security Assurances, Poly 2020, and 6th International Workshop on Data Management and Analytics for Medicine and Healthcare, DMAH 2020

Y2 - 31 August 2020 through 4 September 2020

ER -

Yoon HJ , Klasky HB, Durbin EB, Wu XC, Stroup A, Doherty J et al. Privacy-Preserving Knowledge Transfer with Bootstrap Aggregation of Teacher Ensembles. In Gadepally V, Mattson T, Stonebraker M, Kraska T, Wang F, Luo G, Kong J, Dubovitskaya A, editors, Heterogeneous Data Management, Polystores, and Analytics for Healthcare - VLDB Workshops, Poly 2020 and DMAH 2020, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2021. p. 87-99. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-71055-2_9

Privacy-Preserving Knowledge Transfer with Bootstrap Aggregation of Teacher Ensembles

Abstract

Publication series

Conference

Funding

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this