Integrating apache spark Into PBS-Based HPC environments

Troy Baer; Paul Peltz; Junqi Yin; Edmon Begoli

doi:10.1145/2792745.2792779

Integrating apache spark Into PBS-Based HPC environments

Troy Baer, Paul Peltz, Junqi Yin, Edmon Begoli

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

This paper describes an effort at the University of Tennessee's National Institute for Computational Sciences (NIC- S) to integrate Apache Spark into the widely used TORQUE HPC batch environment. The similarities and differences between the execution of a Spark program and that of an MPI program on a cluster are used to motivate how to implement Spark/TORQUE integration. An implementation of this integration, pbs-spark-submit, is described, including demonstrations of functionality on two HPC clusters and a large shared-memory system.

Original language	English
Title of host publication	Proceedings of the XSEDE 2015 Conference
Subtitle of host publication	Scientific Advancements Enabled by Enhanced Cyberinfrastructure
Publisher	Association for Computing Machinery
ISBN (Electronic)	9781450337205
DOIs	https://doi.org/10.1145/2792745.2792779
State	Published - Jul 26 2015
Externally published	Yes
Event	4th Annual Conference on Extreme Science and Engineering Discovery Environment, XSEDE 2015 - St. Louis, United States Duration: Jul 26 2015 → Jul 30 2015

Publication series

Name	ACM International Conference Proceeding Series
Volume	2015-July

Conference

Conference	4th Annual Conference on Extreme Science and Engineering Discovery Environment, XSEDE 2015
Country/Territory	United States
City	St. Louis
Period	07/26/15 → 07/30/15

Bibliographical note

Publisher Copyright:
Copyright © 2015 ACM.

Keywords

Apache spark
Batch processing
Data analytics
NICS
PBS
TORQUE

Access to Document

10.1145/2792745.2792779

Cite this

Baer, T., Peltz, P., Yin, J., & Begoli, E. (2015). Integrating apache spark Into PBS-Based HPC environments. In Proceedings of the XSEDE 2015 Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure Article a34 (ACM International Conference Proceeding Series; Vol. 2015-July). Association for Computing Machinery. https://doi.org/10.1145/2792745.2792779

@inproceedings{ca86a52462364bfbbd2d020ec859abe1,

title = "Integrating apache spark Into PBS-Based HPC environments",

abstract = "This paper describes an effort at the University of Tennessee's National Institute for Computational Sciences (NIC- S) to integrate Apache Spark into the widely used TORQUE HPC batch environment. The similarities and differences between the execution of a Spark program and that of an MPI program on a cluster are used to motivate how to implement Spark/TORQUE integration. An implementation of this integration, pbs-spark-submit, is described, including demonstrations of functionality on two HPC clusters and a large shared-memory system.",

keywords = "Apache spark, Batch processing, Data analytics, NICS, PBS, TORQUE",

author = "Troy Baer and Paul Peltz and Junqi Yin and Edmon Begoli",

note = "Publisher Copyright: Copyright {\textcopyright} 2015 ACM.; 4th Annual Conference on Extreme Science and Engineering Discovery Environment, XSEDE 2015 ; Conference date: 26-07-2015 Through 30-07-2015",

year = "2015",

month = jul,

day = "26",

doi = "10.1145/2792745.2792779",

language = "English",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

booktitle = "Proceedings of the XSEDE 2015 Conference",

}

Baer, T, Peltz, P, Yin, J & Begoli, E 2015, Integrating apache spark Into PBS-Based HPC environments. in Proceedings of the XSEDE 2015 Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure., a34, ACM International Conference Proceeding Series, vol. 2015-July, Association for Computing Machinery, 4th Annual Conference on Extreme Science and Engineering Discovery Environment, XSEDE 2015, St. Louis, United States, 07/26/15. https://doi.org/10.1145/2792745.2792779

Integrating apache spark Into PBS-Based HPC environments. / Baer, Troy; Peltz, Paul; Yin, Junqi et al.
Proceedings of the XSEDE 2015 Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure. Association for Computing Machinery, 2015. a34 (ACM International Conference Proceeding Series; Vol. 2015-July).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Integrating apache spark Into PBS-Based HPC environments

AU - Baer, Troy

AU - Peltz, Paul

AU - Yin, Junqi

AU - Begoli, Edmon

PY - 2015/7/26

Y1 - 2015/7/26

N2 - This paper describes an effort at the University of Tennessee's National Institute for Computational Sciences (NIC- S) to integrate Apache Spark into the widely used TORQUE HPC batch environment. The similarities and differences between the execution of a Spark program and that of an MPI program on a cluster are used to motivate how to implement Spark/TORQUE integration. An implementation of this integration, pbs-spark-submit, is described, including demonstrations of functionality on two HPC clusters and a large shared-memory system.

AB - This paper describes an effort at the University of Tennessee's National Institute for Computational Sciences (NIC- S) to integrate Apache Spark into the widely used TORQUE HPC batch environment. The similarities and differences between the execution of a Spark program and that of an MPI program on a cluster are used to motivate how to implement Spark/TORQUE integration. An implementation of this integration, pbs-spark-submit, is described, including demonstrations of functionality on two HPC clusters and a large shared-memory system.

KW - Apache spark

KW - Batch processing

KW - Data analytics

KW - NICS

KW - PBS

KW - TORQUE

UR - http://www.scopus.com/inward/record.url?scp=84942802788&partnerID=8YFLogxK

U2 - 10.1145/2792745.2792779

DO - 10.1145/2792745.2792779

M3 - Conference contribution

AN - SCOPUS:84942802788

T3 - ACM International Conference Proceeding Series

BT - Proceedings of the XSEDE 2015 Conference

PB - Association for Computing Machinery

T2 - 4th Annual Conference on Extreme Science and Engineering Discovery Environment, XSEDE 2015

Y2 - 26 July 2015 through 30 July 2015

ER -

Integrating apache spark Into PBS-Based HPC environments

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this