SODA: Science-driven orchestration of data analytics

Jai Dayal; Jay Lofstead; Greg Eisenhauer; Karsten Schwan; Matthew Wolf; Hasan Abbasi; Scott Klasky

doi:10.1109/eScience.2015.59

SODA: Science-driven orchestration of data analytics

Jai Dayal, Jay Lofstead, Greg Eisenhauer, Karsten Schwan, Matthew Wolf, Hasan Abbasi, Scott Klasky

Workflow Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Scopus citations

Abstract

As scientific simulation applications evolve on the path towards exascale, a new model of scientific inquiry is required where concurrently with the running simulation, online analytics operate on the data it produces. By avoiding offline data storage except when absoluately necessary, it enables speeding up the scientific discovery process by providing rapid insights into the simulated science phenomena and affording more frequent, detailed data analytics than is possible with the traditional purely offline approach of using disk for intermediate data storage. However, a challenge for online analytics is to respond to behavior dynamics caused by changing simulation outputs and by unforeseen events on the underlying hardware/software platforms. This paper presents SODA, a set of run-time abstractions for online orchestration of data analytics, realized by embedding analytics tasks into workstations that monitor component behavior and enable responses to run-time changes in their resource demands and in the platform's resource availability. For high end simulations running on a leadership class machine, experimental evaluations show SODA can invoke efficient orchestration operations responding to a diverse set of run-time dynamics at different granularities to meet end-user and analysis specific requirements.

Original language	English
Title of host publication	Proceedings - 11th IEEE International Conference on eScience, eScience 2015
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	475-484
Number of pages	10
ISBN (Electronic)	9781467393256
DOIs	https://doi.org/10.1109/eScience.2015.59
State	Published - Oct 22 2015
Event	11th IEEE International Conference on eScience, eScience 2015 - Munich, Germany Duration: Aug 31 2015 → Sep 4 2015

Publication series

Name	Proceedings - 11th IEEE International Conference on eScience, eScience 2015

Conference

Conference	11th IEEE International Conference on eScience, eScience 2015
Country/Territory	Germany
City	Munich
Period	08/31/15 → 09/4/15

Funding

This research was supported by the Department of Energy Office of Advanced Scientific Computing Research. It also used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. SAND2014-19011 C.

Funders	Funder number
Department of Energy Office of Advanced Scientific Computing Research
U.S. Department of Energy	DE-AC05-00OR22725
Lockheed Martin Corporation
Office of Science
National Nuclear Security Administration	DE-AC04-94AL85000, SAND2014-19011

Keywords

Data analytics
Data staging
In-situ
Resource sharing
Runtime management
Scalable I/O
Visualization

Access to Document

10.1109/eScience.2015.59

Cite this

Dayal, J., Lofstead, J., Eisenhauer, G., Schwan, K., Wolf, M., Abbasi, H., & Klasky, S. (2015). SODA: Science-driven orchestration of data analytics. In Proceedings - 11th IEEE International Conference on eScience, eScience 2015 (pp. 475-484). Article 7304331 (Proceedings - 11th IEEE International Conference on eScience, eScience 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/eScience.2015.59

@inproceedings{3c55f6eb9d094b07beaa0396c694b2e1,

title = "SODA: Science-driven orchestration of data analytics",

abstract = "As scientific simulation applications evolve on the path towards exascale, a new model of scientific inquiry is required where concurrently with the running simulation, online analytics operate on the data it produces. By avoiding offline data storage except when absoluately necessary, it enables speeding up the scientific discovery process by providing rapid insights into the simulated science phenomena and affording more frequent, detailed data analytics than is possible with the traditional purely offline approach of using disk for intermediate data storage. However, a challenge for online analytics is to respond to behavior dynamics caused by changing simulation outputs and by unforeseen events on the underlying hardware/software platforms. This paper presents SODA, a set of run-time abstractions for online orchestration of data analytics, realized by embedding analytics tasks into workstations that monitor component behavior and enable responses to run-time changes in their resource demands and in the platform's resource availability. For high end simulations running on a leadership class machine, experimental evaluations show SODA can invoke efficient orchestration operations responding to a diverse set of run-time dynamics at different granularities to meet end-user and analysis specific requirements.",

keywords = "Data analytics, Data staging, In-situ, Resource sharing, Runtime management, Scalable I/O, Visualization",

author = "Jai Dayal and Jay Lofstead and Greg Eisenhauer and Karsten Schwan and Matthew Wolf and Hasan Abbasi and Scott Klasky",

note = "Publisher Copyright: {\textcopyright} 2015 IEEE.; 11th IEEE International Conference on eScience, eScience 2015 ; Conference date: 31-08-2015 Through 04-09-2015",

year = "2015",

month = oct,

day = "22",

doi = "10.1109/eScience.2015.59",

language = "English",

series = "Proceedings - 11th IEEE International Conference on eScience, eScience 2015",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "475--484",

booktitle = "Proceedings - 11th IEEE International Conference on eScience, eScience 2015",

}

Dayal, J, Lofstead, J, Eisenhauer, G, Schwan, K, Wolf, M, Abbasi, H & Klasky, S 2015, SODA: Science-driven orchestration of data analytics. in Proceedings - 11th IEEE International Conference on eScience, eScience 2015., 7304331, Proceedings - 11th IEEE International Conference on eScience, eScience 2015, Institute of Electrical and Electronics Engineers Inc., pp. 475-484, 11th IEEE International Conference on eScience, eScience 2015, Munich, Germany, 08/31/15. https://doi.org/10.1109/eScience.2015.59

SODA: Science-driven orchestration of data analytics. / Dayal, Jai; Lofstead, Jay; Eisenhauer, Greg et al.
Proceedings - 11th IEEE International Conference on eScience, eScience 2015. Institute of Electrical and Electronics Engineers Inc., 2015. p. 475-484 7304331 (Proceedings - 11th IEEE International Conference on eScience, eScience 2015).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - SODA

T2 - 11th IEEE International Conference on eScience, eScience 2015

AU - Dayal, Jai

AU - Lofstead, Jay

AU - Eisenhauer, Greg

AU - Schwan, Karsten

AU - Wolf, Matthew

AU - Abbasi, Hasan

AU - Klasky, Scott

PY - 2015/10/22

Y1 - 2015/10/22

N2 - As scientific simulation applications evolve on the path towards exascale, a new model of scientific inquiry is required where concurrently with the running simulation, online analytics operate on the data it produces. By avoiding offline data storage except when absoluately necessary, it enables speeding up the scientific discovery process by providing rapid insights into the simulated science phenomena and affording more frequent, detailed data analytics than is possible with the traditional purely offline approach of using disk for intermediate data storage. However, a challenge for online analytics is to respond to behavior dynamics caused by changing simulation outputs and by unforeseen events on the underlying hardware/software platforms. This paper presents SODA, a set of run-time abstractions for online orchestration of data analytics, realized by embedding analytics tasks into workstations that monitor component behavior and enable responses to run-time changes in their resource demands and in the platform's resource availability. For high end simulations running on a leadership class machine, experimental evaluations show SODA can invoke efficient orchestration operations responding to a diverse set of run-time dynamics at different granularities to meet end-user and analysis specific requirements.

AB - As scientific simulation applications evolve on the path towards exascale, a new model of scientific inquiry is required where concurrently with the running simulation, online analytics operate on the data it produces. By avoiding offline data storage except when absoluately necessary, it enables speeding up the scientific discovery process by providing rapid insights into the simulated science phenomena and affording more frequent, detailed data analytics than is possible with the traditional purely offline approach of using disk for intermediate data storage. However, a challenge for online analytics is to respond to behavior dynamics caused by changing simulation outputs and by unforeseen events on the underlying hardware/software platforms. This paper presents SODA, a set of run-time abstractions for online orchestration of data analytics, realized by embedding analytics tasks into workstations that monitor component behavior and enable responses to run-time changes in their resource demands and in the platform's resource availability. For high end simulations running on a leadership class machine, experimental evaluations show SODA can invoke efficient orchestration operations responding to a diverse set of run-time dynamics at different granularities to meet end-user and analysis specific requirements.

KW - Data analytics

KW - Data staging

KW - In-situ

KW - Resource sharing

KW - Runtime management

KW - Scalable I/O

KW - Visualization

UR - http://www.scopus.com/inward/record.url?scp=84959066194&partnerID=8YFLogxK

U2 - 10.1109/eScience.2015.59

DO - 10.1109/eScience.2015.59

M3 - Conference contribution

AN - SCOPUS:84959066194

T3 - Proceedings - 11th IEEE International Conference on eScience, eScience 2015

SP - 475

EP - 484

BT - Proceedings - 11th IEEE International Conference on eScience, eScience 2015

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 31 August 2015 through 4 September 2015

ER -

Dayal J, Lofstead J, Eisenhauer G, Schwan K, Wolf M, Abbasi H et al. SODA: Science-driven orchestration of data analytics. In Proceedings - 11th IEEE International Conference on eScience, eScience 2015. Institute of Electrical and Electronics Engineers Inc. 2015. p. 475-484. 7304331. (Proceedings - 11th IEEE International Conference on eScience, eScience 2015). doi: 10.1109/eScience.2015.59

SODA: Science-driven orchestration of data analytics

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this