A view from ORNL: Scientific data research opportunities in the big data age

Scott Klasky; Matthew Wolf; Mark Ainsworth; Chuck Atkins; Jong Choi; Greg Eisenhauer; Berk Geveci; William Godoy; Mark Kim; James Kress; Tahsin Kurc; Qing Liu; Jeremy Logan; Arthur B. Maccabe; Kshitij Mehta; George Ostrouchov; Manish Parashar; Norbert Podhorszki; David Pugmire; Eric Suchyta; Lipeng Wan; Ruonan Wang

doi:10.1109/ICDCS.2018.00136

A view from ORNL: Scientific data research opportunities in the big data age

Scott Klasky, Matthew Wolf, Mark Ainsworth, Chuck Atkins, Jong Choi, Greg Eisenhauer, Berk Geveci, William Godoy, Mark Kim, James Kress, Tahsin Kurc, Qing Liu, Jeremy Logan, Arthur B. Maccabe, Kshitij Mehta, George Ostrouchov, Manish Parashar, Norbert Podhorszki, David Pugmire, Eric SuchytaLipeng Wan, Ruonan Wang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

11 Scopus citations

Abstract

One of the core issues across computer and computational science today is adapting to, managing, and learning from the influx of 'Big Data'. In the commercial space, this problem has led to a huge investment in new technologies and capabilities that are well adapted to dealing with the sorts of human-generated logs, videos, texts, and other large-data artifacts that are processed and resulted in an explosion of useful platforms and languages (Hadoop, Spark, Pandas, etc.). However, translating this work from the enterprise space to the computational science and HPC community has proven somewhat difficult, in part because of some of the fundamental differences in type and scale of data and timescales surrounding its generation and use. We describe a forward-looking research and development plan which centers around the concept of making Input/Output (I/O) intelligent for users in the scientific community, whether they are accessing scalable storage or performing in situ workflow tasks. Much of our work is based on our experience with the Adaptable I/O System (ADIOS 1.X), and our next generation version of the software ADIOS 2.X [1].

Original language	English
Title of host publication	Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1357-1368
Number of pages	12
ISBN (Electronic)	9781538668719
DOIs	https://doi.org/10.1109/ICDCS.2018.00136
State	Published - Jul 19 2018
Event	38th IEEE International Conference on Distributed Computing Systems, ICDCS 2018 - Vienna, Austria Duration: Jul 2 2018 → Jul 5 2018

Publication series

Name	Proceedings - International Conference on Distributed Computing Systems
Volume	2018-July

Conference

Conference	38th IEEE International Conference on Distributed Computing Systems, ICDCS 2018
Country/Territory	Austria
City	Vienna
Period	07/2/18 → 07/5/18

Funding

Without the continued support from the Department of Energy’s Office of Advanced Scientific Computing Research, the projects upon which this future vision rests would not be possible. Additionally, support from the DOE computing facilities in Oak Ridge and NERSC, as well as the National Science Foundation, were also critical.

Keywords

High Performance Computing
High Performance I/O
In Situ Visualization
Publish/Subscribe

Access to Document

10.1109/ICDCS.2018.00136

Cite this

Klasky, S., Wolf, M., Ainsworth, M., Atkins, C., Choi, J., Eisenhauer, G., Geveci, B., Godoy, W., Kim, M., Kress, J., Kurc, T., Liu, Q., Logan, J., Maccabe, A. B., Mehta, K., Ostrouchov, G., Parashar, M., Podhorszki, N., Pugmire, D., ... Wang, R. (2018). A view from ORNL: Scientific data research opportunities in the big data age. In Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018 (pp. 1357-1368). (Proceedings - International Conference on Distributed Computing Systems; Vol. 2018-July). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICDCS.2018.00136

Klasky, Scott ; Wolf, Matthew ; Ainsworth, Mark et al. / A view from ORNL : Scientific data research opportunities in the big data age. Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 1357-1368 (Proceedings - International Conference on Distributed Computing Systems).

@inproceedings{a9b128f5386d48fba4dc8b15708c917f,

title = "A view from ORNL: Scientific data research opportunities in the big data age",

abstract = "One of the core issues across computer and computational science today is adapting to, managing, and learning from the influx of 'Big Data'. In the commercial space, this problem has led to a huge investment in new technologies and capabilities that are well adapted to dealing with the sorts of human-generated logs, videos, texts, and other large-data artifacts that are processed and resulted in an explosion of useful platforms and languages (Hadoop, Spark, Pandas, etc.). However, translating this work from the enterprise space to the computational science and HPC community has proven somewhat difficult, in part because of some of the fundamental differences in type and scale of data and timescales surrounding its generation and use. We describe a forward-looking research and development plan which centers around the concept of making Input/Output (I/O) intelligent for users in the scientific community, whether they are accessing scalable storage or performing in situ workflow tasks. Much of our work is based on our experience with the Adaptable I/O System (ADIOS 1.X), and our next generation version of the software ADIOS 2.X [1].",

keywords = "High Performance Computing, High Performance I/O, In Situ Visualization, Publish/Subscribe",

author = "Scott Klasky and Matthew Wolf and Mark Ainsworth and Chuck Atkins and Jong Choi and Greg Eisenhauer and Berk Geveci and William Godoy and Mark Kim and James Kress and Tahsin Kurc and Qing Liu and Jeremy Logan and Maccabe, {Arthur B.} and Kshitij Mehta and George Ostrouchov and Manish Parashar and Norbert Podhorszki and David Pugmire and Eric Suchyta and Lipeng Wan and Ruonan Wang",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 38th IEEE International Conference on Distributed Computing Systems, ICDCS 2018 ; Conference date: 02-07-2018 Through 05-07-2018",

year = "2018",

month = jul,

day = "19",

doi = "10.1109/ICDCS.2018.00136",

language = "English",

series = "Proceedings - International Conference on Distributed Computing Systems",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1357--1368",

booktitle = "Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018",

}

Klasky, S, Wolf, M, Ainsworth, M, Atkins, C, Choi, J, Eisenhauer, G, Geveci, B, Godoy, W, Kim, M, Kress, J, Kurc, T, Liu, Q, Logan, J, Maccabe, AB, Mehta, K, Ostrouchov, G, Parashar, M, Podhorszki, N , Pugmire, D , Suchyta, E, Wan, L & Wang, R 2018, A view from ORNL: Scientific data research opportunities in the big data age. in Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018. Proceedings - International Conference on Distributed Computing Systems, vol. 2018-July, Institute of Electrical and Electronics Engineers Inc., pp. 1357-1368, 38th IEEE International Conference on Distributed Computing Systems, ICDCS 2018, Vienna, Austria, 07/2/18. https://doi.org/10.1109/ICDCS.2018.00136

A view from ORNL: Scientific data research opportunities in the big data age. / Klasky, Scott; Wolf, Matthew; Ainsworth, Mark et al.
Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 1357-1368 (Proceedings - International Conference on Distributed Computing Systems; Vol. 2018-July).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A view from ORNL

T2 - 38th IEEE International Conference on Distributed Computing Systems, ICDCS 2018

AU - Klasky, Scott

AU - Wolf, Matthew

AU - Ainsworth, Mark

AU - Atkins, Chuck

AU - Choi, Jong

AU - Eisenhauer, Greg

AU - Geveci, Berk

AU - Godoy, William

AU - Kim, Mark

AU - Kress, James

AU - Kurc, Tahsin

AU - Liu, Qing

AU - Logan, Jeremy

AU - Maccabe, Arthur B.

AU - Mehta, Kshitij

AU - Ostrouchov, George

AU - Parashar, Manish

AU - Podhorszki, Norbert

AU - Pugmire, David

AU - Suchyta, Eric

AU - Wan, Lipeng

AU - Wang, Ruonan

PY - 2018/7/19

Y1 - 2018/7/19

N2 - One of the core issues across computer and computational science today is adapting to, managing, and learning from the influx of 'Big Data'. In the commercial space, this problem has led to a huge investment in new technologies and capabilities that are well adapted to dealing with the sorts of human-generated logs, videos, texts, and other large-data artifacts that are processed and resulted in an explosion of useful platforms and languages (Hadoop, Spark, Pandas, etc.). However, translating this work from the enterprise space to the computational science and HPC community has proven somewhat difficult, in part because of some of the fundamental differences in type and scale of data and timescales surrounding its generation and use. We describe a forward-looking research and development plan which centers around the concept of making Input/Output (I/O) intelligent for users in the scientific community, whether they are accessing scalable storage or performing in situ workflow tasks. Much of our work is based on our experience with the Adaptable I/O System (ADIOS 1.X), and our next generation version of the software ADIOS 2.X [1].

AB - One of the core issues across computer and computational science today is adapting to, managing, and learning from the influx of 'Big Data'. In the commercial space, this problem has led to a huge investment in new technologies and capabilities that are well adapted to dealing with the sorts of human-generated logs, videos, texts, and other large-data artifacts that are processed and resulted in an explosion of useful platforms and languages (Hadoop, Spark, Pandas, etc.). However, translating this work from the enterprise space to the computational science and HPC community has proven somewhat difficult, in part because of some of the fundamental differences in type and scale of data and timescales surrounding its generation and use. We describe a forward-looking research and development plan which centers around the concept of making Input/Output (I/O) intelligent for users in the scientific community, whether they are accessing scalable storage or performing in situ workflow tasks. Much of our work is based on our experience with the Adaptable I/O System (ADIOS 1.X), and our next generation version of the software ADIOS 2.X [1].

KW - High Performance Computing

KW - High Performance I/O

KW - In Situ Visualization

KW - Publish/Subscribe

UR - http://www.scopus.com/inward/record.url?scp=85050963067&partnerID=8YFLogxK

U2 - 10.1109/ICDCS.2018.00136

DO - 10.1109/ICDCS.2018.00136

M3 - Conference contribution

AN - SCOPUS:85050963067

T3 - Proceedings - International Conference on Distributed Computing Systems

SP - 1357

EP - 1368

BT - Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 2 July 2018 through 5 July 2018

ER -

Klasky S, Wolf M, Ainsworth M, Atkins C, Choi J, Eisenhauer G et al. A view from ORNL: Scientific data research opportunities in the big data age. In Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 1357-1368. (Proceedings - International Conference on Distributed Computing Systems). doi: 10.1109/ICDCS.2018.00136

A view from ORNL: Scientific data research opportunities in the big data age

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this