Automated integration of continental-scale observations in near-real time for simulation and analysis of biosphere-atmosphere interactions

David J. Durden; Stefan Metzger; Housen Chu; Nathan Collier; Kenneth J. Davis; Ankur R. Desai; Jitendra Kumar; William R. Wieder; Min Xu; Forrest M. Hoffman

doi:10.1007/978-3-030-63393-6_14

Automated integration of continental-scale observations in near-real time for simulation and analysis of biosphere-atmosphere interactions

David J. Durden, Stefan Metzger, Housen Chu, Nathan Collier, Kenneth J. Davis, Ankur R. Desai, Jitendra Kumar, William R. Wieder, Min Xu, Forrest M. Hoffman

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

The National Ecological Observatory Network (NEON) is a continental-scale observatory with sites across the US collecting standardized ecological observations that will operate for multiple decades. To maximize the utility of NEON data, we envision edge computing systems that gather, calibrate, aggregate, and ingest measurements in an integrated fashion. Edge systems will employ machine learning methods to cross-calibrate, gap-fill and provision data in near-real time to the NEON Data Portal and to High Performance Computing (HPC) systems, running ensembles of Earth system models (ESMs) that assimilate the data. For the first time gridded EC data products and response functions promise to offset pervasive observational biases through evaluating, benchmarking, optimizing parameters, and training new machine learning parameterizations within ESMs all at the same model-grid scale. Leveraging open-source software for EC data analysis, we are already building software infrastructure for integration of near-real time data streams into the International Land Model Benchmarking (ILAMB) package for use by the wider research community. We will present a perspective on the design and integration of end-to-end infrastructure for data acquisition, edge computing, HPC simulation, analysis, and validation, where Artificial Intelligence (AI) approaches are used throughout the distributed workflow to improve accuracy and computational performance.

Original language	English
Title of host publication	Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers
Editors	Jeffrey Nichols, Arthur ‘Barney’ Maccabe, Suzanne Parete-Koon, Becky Verastegui, Oscar Hernandez, Theresa Ahearn
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	204-225
Number of pages	22
ISBN (Print)	9783030633929
DOIs	https://doi.org/10.1007/978-3-030-63393-6_14
State	Published - 2021
Event	17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020 - Virtual, Online Duration: Aug 26 2020 → Aug 28 2020

Publication series

Name	Communications in Computer and Information Science
Volume	1315 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020
City	Virtual, Online
Period	08/26/20 → 08/28/20

Funding

Acknowledgements. The National Ecological Observatory Network is a project sponsored by the National Science Foundation and managed under cooperative agreement by Battelle. Some of this material is based upon work supported by the National Science Foundation (Grant DBI-0752017). We acknowledge contributions for some of this material from participants of NSF-sponsored joint NCAR and NEON workshop, Predicting life in the Earth system – linking the geosciences and ecology, and the continued efforts in post-workshop working groups. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. Portions of this research were sponsored by the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory and used resources of the Oak Ridge Leadership Computing Facility (OLCF) at Oak Ridge National Laboratory, which is managed by UT-Battelle, LLC, for the U.S. Department of Energy under contract DE–AC05–00OR22725. Additional support was provided by the Reducing Uncertainties in Biogeochemical Interactions through Synthesis and Computation Science Focus Area (RUBISCO SFA), which is sponsored by the Regional and Global Model Analysis (RGMA) activity of the Earth & Environmental System Modeling (EESM) Program in the Earth and Environmental Systems Sciences Division (EESSD) of the Office of Biological and Environmental Research (BER) in the U.S. Department of Energy Office of Science. Some contributions to this research were supported by NASA’s Earth Science Division (Grant NNX15AG76G). We acknowledge contributions for some of this material from all participants in the RUBISCO-AmeriFlux Working Group, which is supported by the RUBISCO SFA.

Keywords

Data-model integration
Earth system models · Land surface models
Eddy-covariance
Edge computing systems
Environmental observatory
High performance computing
International Land Model Benchmarking (ILAMB)
Model benchmarking
National Ecological Observatory Network (NEON)

Access to Document

10.1007/978-3-030-63393-6_14

Cite this

Durden, D. J., Metzger, S., Chu, H., Collier, N., Davis, K. J., Desai, A. R., Kumar, J., Wieder, W. R., Xu, M., & Hoffman, F. M. (2021). Automated integration of continental-scale observations in near-real time for simulation and analysis of biosphere-atmosphere interactions. In J. Nichols, A. B. Maccabe, S. Parete-Koon, B. Verastegui, O. Hernandez, & T. Ahearn (Eds.), Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers (pp. 204-225). (Communications in Computer and Information Science; Vol. 1315 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-63393-6_14

Durden, David J. ; Metzger, Stefan ; Chu, Housen et al. / Automated integration of continental-scale observations in near-real time for simulation and analysis of biosphere-atmosphere interactions. Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers. editor / Jeffrey Nichols ; Arthur ‘Barney’ Maccabe ; Suzanne Parete-Koon ; Becky Verastegui ; Oscar Hernandez ; Theresa Ahearn. Springer Science and Business Media Deutschland GmbH, 2021. pp. 204-225 (Communications in Computer and Information Science).

@inproceedings{b3c90707453442b888c8dc364cc346a2,

title = "Automated integration of continental-scale observations in near-real time for simulation and analysis of biosphere-atmosphere interactions",

abstract = "The National Ecological Observatory Network (NEON) is a continental-scale observatory with sites across the US collecting standardized ecological observations that will operate for multiple decades. To maximize the utility of NEON data, we envision edge computing systems that gather, calibrate, aggregate, and ingest measurements in an integrated fashion. Edge systems will employ machine learning methods to cross-calibrate, gap-fill and provision data in near-real time to the NEON Data Portal and to High Performance Computing (HPC) systems, running ensembles of Earth system models (ESMs) that assimilate the data. For the first time gridded EC data products and response functions promise to offset pervasive observational biases through evaluating, benchmarking, optimizing parameters, and training new machine learning parameterizations within ESMs all at the same model-grid scale. Leveraging open-source software for EC data analysis, we are already building software infrastructure for integration of near-real time data streams into the International Land Model Benchmarking (ILAMB) package for use by the wider research community. We will present a perspective on the design and integration of end-to-end infrastructure for data acquisition, edge computing, HPC simulation, analysis, and validation, where Artificial Intelligence (AI) approaches are used throughout the distributed workflow to improve accuracy and computational performance.",

keywords = "Data-model integration, Earth system models · Land surface models, Eddy-covariance, Edge computing systems, Environmental observatory, High performance computing, International Land Model Benchmarking (ILAMB), Model benchmarking, National Ecological Observatory Network (NEON)",

author = "Durden, \{David J.\} and Stefan Metzger and Housen Chu and Nathan Collier and Davis, \{Kenneth J.\} and Desai, \{Ankur R.\} and Jitendra Kumar and Wieder, \{William R.\} and Min Xu and Hoffman, \{Forrest M.\}",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2020.; 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020 ; Conference date: 26-08-2020 Through 28-08-2020",

year = "2021",

doi = "10.1007/978-3-030-63393-6\_14",

language = "English",

isbn = "9783030633929",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "204--225",

editor = "Jeffrey Nichols and Maccabe, \{Arthur {\textquoteleft}Barney{\textquoteright}\} and Suzanne Parete-Koon and Becky Verastegui and Oscar Hernandez and Theresa Ahearn",

booktitle = "Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers",

}

Durden, DJ, Metzger, S, Chu, H, Collier, N, Davis, KJ, Desai, AR, Kumar, J, Wieder, WR, Xu, M & Hoffman, FM 2021, Automated integration of continental-scale observations in near-real time for simulation and analysis of biosphere-atmosphere interactions. in J Nichols, AB Maccabe, S Parete-Koon, B Verastegui, O Hernandez & T Ahearn (eds), Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers. Communications in Computer and Information Science, vol. 1315 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 204-225, 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Virtual, Online, 08/26/20. https://doi.org/10.1007/978-3-030-63393-6_14

Automated integration of continental-scale observations in near-real time for simulation and analysis of biosphere-atmosphere interactions. / Durden, David J.; Metzger, Stefan; Chu, Housen et al.
Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers. ed. / Jeffrey Nichols; Arthur ‘Barney’ Maccabe; Suzanne Parete-Koon; Becky Verastegui; Oscar Hernandez; Theresa Ahearn. Springer Science and Business Media Deutschland GmbH, 2021. p. 204-225 (Communications in Computer and Information Science; Vol. 1315 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Automated integration of continental-scale observations in near-real time for simulation and analysis of biosphere-atmosphere interactions

AU - Durden, David J.

AU - Metzger, Stefan

AU - Chu, Housen

AU - Collier, Nathan

AU - Davis, Kenneth J.

AU - Desai, Ankur R.

AU - Kumar, Jitendra

AU - Wieder, William R.

AU - Xu, Min

AU - Hoffman, Forrest M.

N1 - Publisher Copyright: © Springer Nature Switzerland AG 2020.

PY - 2021

Y1 - 2021

N2 - The National Ecological Observatory Network (NEON) is a continental-scale observatory with sites across the US collecting standardized ecological observations that will operate for multiple decades. To maximize the utility of NEON data, we envision edge computing systems that gather, calibrate, aggregate, and ingest measurements in an integrated fashion. Edge systems will employ machine learning methods to cross-calibrate, gap-fill and provision data in near-real time to the NEON Data Portal and to High Performance Computing (HPC) systems, running ensembles of Earth system models (ESMs) that assimilate the data. For the first time gridded EC data products and response functions promise to offset pervasive observational biases through evaluating, benchmarking, optimizing parameters, and training new machine learning parameterizations within ESMs all at the same model-grid scale. Leveraging open-source software for EC data analysis, we are already building software infrastructure for integration of near-real time data streams into the International Land Model Benchmarking (ILAMB) package for use by the wider research community. We will present a perspective on the design and integration of end-to-end infrastructure for data acquisition, edge computing, HPC simulation, analysis, and validation, where Artificial Intelligence (AI) approaches are used throughout the distributed workflow to improve accuracy and computational performance.

AB - The National Ecological Observatory Network (NEON) is a continental-scale observatory with sites across the US collecting standardized ecological observations that will operate for multiple decades. To maximize the utility of NEON data, we envision edge computing systems that gather, calibrate, aggregate, and ingest measurements in an integrated fashion. Edge systems will employ machine learning methods to cross-calibrate, gap-fill and provision data in near-real time to the NEON Data Portal and to High Performance Computing (HPC) systems, running ensembles of Earth system models (ESMs) that assimilate the data. For the first time gridded EC data products and response functions promise to offset pervasive observational biases through evaluating, benchmarking, optimizing parameters, and training new machine learning parameterizations within ESMs all at the same model-grid scale. Leveraging open-source software for EC data analysis, we are already building software infrastructure for integration of near-real time data streams into the International Land Model Benchmarking (ILAMB) package for use by the wider research community. We will present a perspective on the design and integration of end-to-end infrastructure for data acquisition, edge computing, HPC simulation, analysis, and validation, where Artificial Intelligence (AI) approaches are used throughout the distributed workflow to improve accuracy and computational performance.

KW - Data-model integration

KW - Earth system models · Land surface models

KW - Eddy-covariance

KW - Edge computing systems

KW - Environmental observatory

KW - High performance computing

KW - International Land Model Benchmarking (ILAMB)

KW - Model benchmarking

KW - National Ecological Observatory Network (NEON)

UR - http://www.scopus.com/inward/record.url?scp=85107307710&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-63393-6_14

DO - 10.1007/978-3-030-63393-6_14

M3 - Conference contribution

AN - SCOPUS:85107307710

SN - 9783030633929

T3 - Communications in Computer and Information Science

SP - 204

EP - 225

BT - Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers

A2 - Nichols, Jeffrey

A2 - Maccabe, Arthur ‘Barney’

A2 - Parete-Koon, Suzanne

A2 - Verastegui, Becky

A2 - Hernandez, Oscar

A2 - Ahearn, Theresa

PB - Springer Science and Business Media Deutschland GmbH

T2 - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020

Y2 - 26 August 2020 through 28 August 2020

ER -

Durden DJ, Metzger S, Chu H, Collier N, Davis KJ, Desai AR et al. Automated integration of continental-scale observations in near-real time for simulation and analysis of biosphere-atmosphere interactions. In Nichols J, Maccabe AB, Parete-Koon S, Verastegui B, Hernandez O, Ahearn T, editors, Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2021. p. 204-225. (Communications in Computer and Information Science). doi: 10.1007/978-3-030-63393-6_14

Automated integration of continental-scale observations in near-real time for simulation and analysis of biosphere-atmosphere interactions

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this