The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data

Luca Cinquini; Daniel Crichton; Chris Mattmann; John Harney; Galen Shipman; Feiyi Wang; Rachana Ananthakrishnan; Neill Miller; Sebastian Denvil; Mark Morgan; Zed Pobre; Gavin M. Bell; Charles Doutriaux; Robert Drach; Dean Williams; Philip Kershaw; Stephen Pascoe; Estanislao Gonzalez; Sandro Fiore; Roland Schweitzer

doi:10.1016/j.future.2013.07.002

The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data

Luca Cinquini, Daniel Crichton, Chris Mattmann, John Harney, Galen Shipman, Feiyi Wang, Rachana Ananthakrishnan, Neill Miller, Sebastian Denvil, Mark Morgan, Zed Pobre, Gavin M. Bell, Charles Doutriaux, Robert Drach, Dean Williams, Philip Kershaw, Stephen Pascoe, Estanislao Gonzalez, Sandro Fiore, Roland Schweitzer

Analytics and AI Methods at Scale

Research output: Contribution to journal › Article › peer-review

159 Scopus citations

Abstract

The Earth System Grid Federation (ESGF) is a multi-agency, international collaboration that aims at developing the software infrastructure needed to facilitate and empower the study of climate change on a global scale. The ESGF's architecture employs a system of geographically distributed peer nodes, which are independently administered yet united by the adoption of common federation protocols and application programming interfaces (APIs). The cornerstones of its interoperability are the peer-to-peer messaging that is continuously exchanged among all nodes in the federation; a shared architecture and API for search and discovery; and a security infrastructure based on industry standards (OpenID, SSL, GSI and SAML). The ESGF software stack integrates custom components (for data publishing, searching, user interface, security and messaging), developed collaboratively by the team, with popular application engines (Tomcat, Solr) available from the open source community. The full ESGF infrastructure has now been adopted by multiple Earth science projects and allows access to petabytes of geophysical data, including the entire Fifth Coupled Model Intercomparison Project (CMIP5) output used by the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and a suite of satellite observations (obs4MIPs) and reanalysis data sets (ANA4MIPs). This paper presents ESGF as a successful example of integration of disparate open source technologies into a cohesive, wide functional system, and describes our experience in building and operating a distributed and federated infrastructure to serve the needs of the global climate science community.

Original language	English
Pages (from-to)	400-417
Number of pages	18
Journal	Future Generation Computer Systems
Volume	36
DOIs	https://doi.org/10.1016/j.future.2013.07.002
State	Published - Jul 2014

Funding

The development and operation of ESGF is supported by the efforts of principal investigators, software engineers, data managers and system administrators from many agencies and institutions worldwide. Primary contributors include ANL, ANU, BADC, CMCC, DKRZ, ESRL, GFDL, GSFC, JPL, IPSL, NCAR, ORNL, LBNL, LLNL (leading institution), PMEL, PNNL and SNL. Major funding provided by the U.S. Department of Energy , the National Atmospheric and Space Administration (NASA) , and the European Infrastructure for the European Network for Earth System Modeling (IS-ENES) .

Funders	Funder number
European Infrastructure for the European Network for Earth System Modeling
IS-ENES
U.S. Department of Energy
National Oceanic and Atmospheric Administration

Keywords

CMIP5
Climate science
Discovery
Federation
Peer-to-peer
Search

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1016/j.future.2013.07.002

Cite this

Cinquini, L., Crichton, D., Mattmann, C., Harney, J., Shipman, G., Wang, F., Ananthakrishnan, R., Miller, N., Denvil, S., Morgan, M., Pobre, Z., Bell, G. M., Doutriaux, C., Drach, R., Williams, D., Kershaw, P., Pascoe, S., Gonzalez, E., Fiore, S., & Schweitzer, R. (2014). The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data. Future Generation Computer Systems, 36, 400-417. https://doi.org/10.1016/j.future.2013.07.002

@article{2c6fe56f5342467bb435b5e9f3439f8a,

title = "The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data",

abstract = "The Earth System Grid Federation (ESGF) is a multi-agency, international collaboration that aims at developing the software infrastructure needed to facilitate and empower the study of climate change on a global scale. The ESGF's architecture employs a system of geographically distributed peer nodes, which are independently administered yet united by the adoption of common federation protocols and application programming interfaces (APIs). The cornerstones of its interoperability are the peer-to-peer messaging that is continuously exchanged among all nodes in the federation; a shared architecture and API for search and discovery; and a security infrastructure based on industry standards (OpenID, SSL, GSI and SAML). The ESGF software stack integrates custom components (for data publishing, searching, user interface, security and messaging), developed collaboratively by the team, with popular application engines (Tomcat, Solr) available from the open source community. The full ESGF infrastructure has now been adopted by multiple Earth science projects and allows access to petabytes of geophysical data, including the entire Fifth Coupled Model Intercomparison Project (CMIP5) output used by the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and a suite of satellite observations (obs4MIPs) and reanalysis data sets (ANA4MIPs). This paper presents ESGF as a successful example of integration of disparate open source technologies into a cohesive, wide functional system, and describes our experience in building and operating a distributed and federated infrastructure to serve the needs of the global climate science community.",

keywords = "CMIP5, Climate science, Discovery, Federation, Peer-to-peer, Search",

author = "Luca Cinquini and Daniel Crichton and Chris Mattmann and John Harney and Galen Shipman and Feiyi Wang and Rachana Ananthakrishnan and Neill Miller and Sebastian Denvil and Mark Morgan and Zed Pobre and Bell, {Gavin M.} and Charles Doutriaux and Robert Drach and Dean Williams and Philip Kershaw and Stephen Pascoe and Estanislao Gonzalez and Sandro Fiore and Roland Schweitzer",

year = "2014",

month = jul,

doi = "10.1016/j.future.2013.07.002",

language = "English",

volume = "36",

pages = "400--417",

journal = "Future Generation Computer Systems",

issn = "0167-739X",

publisher = "Elsevier",

}

Cinquini, L, Crichton, D, Mattmann, C, Harney, J, Shipman, G, Wang, F, Ananthakrishnan, R, Miller, N, Denvil, S, Morgan, M, Pobre, Z, Bell, GM, Doutriaux, C, Drach, R, Williams, D, Kershaw, P, Pascoe, S, Gonzalez, E, Fiore, S & Schweitzer, R 2014, 'The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data', Future Generation Computer Systems, vol. 36, pp. 400-417. https://doi.org/10.1016/j.future.2013.07.002

TY - JOUR

T1 - The Earth System Grid Federation

T2 - An open infrastructure for access to distributed geospatial data

AU - Cinquini, Luca

AU - Crichton, Daniel

AU - Mattmann, Chris

AU - Harney, John

AU - Shipman, Galen

AU - Wang, Feiyi

AU - Ananthakrishnan, Rachana

AU - Miller, Neill

AU - Denvil, Sebastian

AU - Morgan, Mark

AU - Pobre, Zed

AU - Bell, Gavin M.

AU - Doutriaux, Charles

AU - Drach, Robert

AU - Williams, Dean

AU - Kershaw, Philip

AU - Pascoe, Stephen

AU - Gonzalez, Estanislao

AU - Fiore, Sandro

AU - Schweitzer, Roland

PY - 2014/7

Y1 - 2014/7

N2 - The Earth System Grid Federation (ESGF) is a multi-agency, international collaboration that aims at developing the software infrastructure needed to facilitate and empower the study of climate change on a global scale. The ESGF's architecture employs a system of geographically distributed peer nodes, which are independently administered yet united by the adoption of common federation protocols and application programming interfaces (APIs). The cornerstones of its interoperability are the peer-to-peer messaging that is continuously exchanged among all nodes in the federation; a shared architecture and API for search and discovery; and a security infrastructure based on industry standards (OpenID, SSL, GSI and SAML). The ESGF software stack integrates custom components (for data publishing, searching, user interface, security and messaging), developed collaboratively by the team, with popular application engines (Tomcat, Solr) available from the open source community. The full ESGF infrastructure has now been adopted by multiple Earth science projects and allows access to petabytes of geophysical data, including the entire Fifth Coupled Model Intercomparison Project (CMIP5) output used by the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and a suite of satellite observations (obs4MIPs) and reanalysis data sets (ANA4MIPs). This paper presents ESGF as a successful example of integration of disparate open source technologies into a cohesive, wide functional system, and describes our experience in building and operating a distributed and federated infrastructure to serve the needs of the global climate science community.

AB - The Earth System Grid Federation (ESGF) is a multi-agency, international collaboration that aims at developing the software infrastructure needed to facilitate and empower the study of climate change on a global scale. The ESGF's architecture employs a system of geographically distributed peer nodes, which are independently administered yet united by the adoption of common federation protocols and application programming interfaces (APIs). The cornerstones of its interoperability are the peer-to-peer messaging that is continuously exchanged among all nodes in the federation; a shared architecture and API for search and discovery; and a security infrastructure based on industry standards (OpenID, SSL, GSI and SAML). The ESGF software stack integrates custom components (for data publishing, searching, user interface, security and messaging), developed collaboratively by the team, with popular application engines (Tomcat, Solr) available from the open source community. The full ESGF infrastructure has now been adopted by multiple Earth science projects and allows access to petabytes of geophysical data, including the entire Fifth Coupled Model Intercomparison Project (CMIP5) output used by the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and a suite of satellite observations (obs4MIPs) and reanalysis data sets (ANA4MIPs). This paper presents ESGF as a successful example of integration of disparate open source technologies into a cohesive, wide functional system, and describes our experience in building and operating a distributed and federated infrastructure to serve the needs of the global climate science community.

KW - CMIP5

KW - Climate science

KW - Discovery

KW - Federation

KW - Peer-to-peer

KW - Search

UR - http://www.scopus.com/inward/record.url?scp=84899626374&partnerID=8YFLogxK

U2 - 10.1016/j.future.2013.07.002

DO - 10.1016/j.future.2013.07.002

M3 - Article

AN - SCOPUS:84899626374

SN - 0167-739X

VL - 36

SP - 400

EP - 417

JO - Future Generation Computer Systems

JF - Future Generation Computer Systems

ER -

The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data

Abstract

Funding

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this