In-situ I/O processing: A case for location flexibility

Fang Zheng; Hasan Abbasi; Jianting Cao; Jai Dayal; Karsten Schwan; Matthew Wolf; Scott Klasky; Norbert Podhorszki

doi:10.1145/2159352.2159362

In-situ I/O processing: A case for location flexibility

Fang Zheng, Hasan Abbasi, Jianting Cao, Jai Dayal, Karsten Schwan, Matthew Wolf, Scott Klasky, Norbert Podhorszki

Workflow Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

12 Scopus citations

Abstract

Increasingly severe I/O bottlenecks on High-End Computing machines are prompting scientists to process output data during simulation time, "in-situ", and before placing data on disks. This paper argues for flexibility in the implementation of such in-situ data analytics, using measurements and a performance model that demonstrate the potential advantages and limitations of performing analytics at different levels of the I/O hierarchy, including on a machine's compute nodes vs. on separate "staging" nodes dedicated to analysis tasks. Model and measurement results are guided by realistic large-scale applications running on leadership class machines, and I/O and analytics actions are described as computational dataflow graphs - termed I/O graphs - that combine data movement with 'in transit' operations on data as it is being moved across the I/O hierarchy. Results demonstrate the importance of flexibility in analytics placement and characterize the attributes of analytics operations that lead to different placement decisions.

Original language	English
Title of host publication	PDSW'11 - Proceedings of the 6th Parallel Data Storage Workshop, Co-located with SC'11
Pages	37-42
Number of pages	6
DOIs	https://doi.org/10.1145/2159352.2159362
State	Published - 2011
Event	6th Parallel Data Storage Workshop, PDSW'11, Co-located with SC'11 - Seattle, WA, United States Duration: Nov 13 2011 → Nov 13 2011

Publication series

Name	PDSW'11 - Proceedings of the 6th Parallel Data Storage Workshop, Co-located with SC'11

Conference

Conference	6th Parallel Data Storage Workshop, PDSW'11, Co-located with SC'11
Country/Territory	United States
City	Seattle, WA
Period	11/13/11 → 11/13/11

Keywords

I/O
analytics
in-situ processing
placement
staging

Access to Document

10.1145/2159352.2159362

Cite this

Zheng, F., Abbasi, H., Cao, J., Dayal, J., Schwan, K., Wolf, M., Klasky, S., & Podhorszki, N. (2011). In-situ I/O processing: A case for location flexibility. In PDSW'11 - Proceedings of the 6th Parallel Data Storage Workshop, Co-located with SC'11 (pp. 37-42). (PDSW'11 - Proceedings of the 6th Parallel Data Storage Workshop, Co-located with SC'11). https://doi.org/10.1145/2159352.2159362

@inproceedings{801d8cf7fcda4390ae94e8ea716f3a33,

title = "In-situ I/O processing: A case for location flexibility",

abstract = "Increasingly severe I/O bottlenecks on High-End Computing machines are prompting scientists to process output data during simulation time, {"}in-situ{"}, and before placing data on disks. This paper argues for flexibility in the implementation of such in-situ data analytics, using measurements and a performance model that demonstrate the potential advantages and limitations of performing analytics at different levels of the I/O hierarchy, including on a machine's compute nodes vs. on separate {"}staging{"} nodes dedicated to analysis tasks. Model and measurement results are guided by realistic large-scale applications running on leadership class machines, and I/O and analytics actions are described as computational dataflow graphs - termed I/O graphs - that combine data movement with 'in transit' operations on data as it is being moved across the I/O hierarchy. Results demonstrate the importance of flexibility in analytics placement and characterize the attributes of analytics operations that lead to different placement decisions.",

keywords = "I/O, analytics, in-situ processing, placement, staging",

author = "Fang Zheng and Hasan Abbasi and Jianting Cao and Jai Dayal and Karsten Schwan and Matthew Wolf and Scott Klasky and Norbert Podhorszki",

year = "2011",

doi = "10.1145/2159352.2159362",

language = "English",

isbn = "9781450311038",

series = "PDSW'11 - Proceedings of the 6th Parallel Data Storage Workshop, Co-located with SC'11",

pages = "37--42",

booktitle = "PDSW'11 - Proceedings of the 6th Parallel Data Storage Workshop, Co-located with SC'11",

note = "6th Parallel Data Storage Workshop, PDSW'11, Co-located with SC'11 ; Conference date: 13-11-2011 Through 13-11-2011",

}

Zheng, F, Abbasi, H, Cao, J, Dayal, J, Schwan, K, Wolf, M, Klasky, S & Podhorszki, N 2011, In-situ I/O processing: A case for location flexibility. in PDSW'11 - Proceedings of the 6th Parallel Data Storage Workshop, Co-located with SC'11. PDSW'11 - Proceedings of the 6th Parallel Data Storage Workshop, Co-located with SC'11, pp. 37-42, 6th Parallel Data Storage Workshop, PDSW'11, Co-located with SC'11, Seattle, WA, United States, 11/13/11. https://doi.org/10.1145/2159352.2159362

In-situ I/O processing: A case for location flexibility. / Zheng, Fang; Abbasi, Hasan; Cao, Jianting et al.
PDSW'11 - Proceedings of the 6th Parallel Data Storage Workshop, Co-located with SC'11. 2011. p. 37-42 (PDSW'11 - Proceedings of the 6th Parallel Data Storage Workshop, Co-located with SC'11).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - In-situ I/O processing

T2 - 6th Parallel Data Storage Workshop, PDSW'11, Co-located with SC'11

AU - Zheng, Fang

AU - Abbasi, Hasan

AU - Cao, Jianting

AU - Dayal, Jai

AU - Schwan, Karsten

AU - Wolf, Matthew

AU - Klasky, Scott

AU - Podhorszki, Norbert

PY - 2011

Y1 - 2011

N2 - Increasingly severe I/O bottlenecks on High-End Computing machines are prompting scientists to process output data during simulation time, "in-situ", and before placing data on disks. This paper argues for flexibility in the implementation of such in-situ data analytics, using measurements and a performance model that demonstrate the potential advantages and limitations of performing analytics at different levels of the I/O hierarchy, including on a machine's compute nodes vs. on separate "staging" nodes dedicated to analysis tasks. Model and measurement results are guided by realistic large-scale applications running on leadership class machines, and I/O and analytics actions are described as computational dataflow graphs - termed I/O graphs - that combine data movement with 'in transit' operations on data as it is being moved across the I/O hierarchy. Results demonstrate the importance of flexibility in analytics placement and characterize the attributes of analytics operations that lead to different placement decisions.

AB - Increasingly severe I/O bottlenecks on High-End Computing machines are prompting scientists to process output data during simulation time, "in-situ", and before placing data on disks. This paper argues for flexibility in the implementation of such in-situ data analytics, using measurements and a performance model that demonstrate the potential advantages and limitations of performing analytics at different levels of the I/O hierarchy, including on a machine's compute nodes vs. on separate "staging" nodes dedicated to analysis tasks. Model and measurement results are guided by realistic large-scale applications running on leadership class machines, and I/O and analytics actions are described as computational dataflow graphs - termed I/O graphs - that combine data movement with 'in transit' operations on data as it is being moved across the I/O hierarchy. Results demonstrate the importance of flexibility in analytics placement and characterize the attributes of analytics operations that lead to different placement decisions.

KW - I/O

KW - analytics

KW - in-situ processing

KW - placement

KW - staging

UR - http://www.scopus.com/inward/record.url?scp=84859847646&partnerID=8YFLogxK

U2 - 10.1145/2159352.2159362

DO - 10.1145/2159352.2159362

M3 - Conference contribution

AN - SCOPUS:84859847646

SN - 9781450311038

T3 - PDSW'11 - Proceedings of the 6th Parallel Data Storage Workshop, Co-located with SC'11

SP - 37

EP - 42

BT - PDSW'11 - Proceedings of the 6th Parallel Data Storage Workshop, Co-located with SC'11

Y2 - 13 November 2011 through 13 November 2011

ER -

In-situ I/O processing: A case for location flexibility

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this