TY - GEN
T1 - Interlanguage parallel scripting for distributed-memory scientific computing
AU - Wozniak, Justin M.
AU - Armstrong, Timothy G.
AU - Maheshwari, Ketan
AU - Katz, Daniel S.
AU - Wilde, Michael
AU - Foster, Ian T.
N1 - Publisher Copyright:
© 2015 ACM.
PY - 2015/11/15
Y1 - 2015/11/15
N2 - Scripting languages such as Python and R have been widely adopted as tools for the development of scientific software because of the expressiveness of the languages and their available libraries. However, deploying scripted applications on large-scale parallel computer systems such as the IBM Blue Gene/Q or Cray XE6 is a challenge because of issues including operating system limitations, interoperabil- ity challenges, and parallel filesystem overheads due to the small file system accesses common in scripted approaches. We present a new approach to these problems in which the Swift scripting system is used to integrate high-level scripts written in Python, R, and Tcl with native code developed in C, C++, and Fortran, by linking Swift to the library interfaces to the script interpreters. We present a technique to eciently launch scripted applications on supercomputers, and we demonstrate high performance, such as invoking 14M Python interpreters per second on Blue Waters.
AB - Scripting languages such as Python and R have been widely adopted as tools for the development of scientific software because of the expressiveness of the languages and their available libraries. However, deploying scripted applications on large-scale parallel computer systems such as the IBM Blue Gene/Q or Cray XE6 is a challenge because of issues including operating system limitations, interoperabil- ity challenges, and parallel filesystem overheads due to the small file system accesses common in scripted approaches. We present a new approach to these problems in which the Swift scripting system is used to integrate high-level scripts written in Python, R, and Tcl with native code developed in C, C++, and Fortran, by linking Swift to the library interfaces to the script interpreters. We present a technique to eciently launch scripted applications on supercomputers, and we demonstrate high performance, such as invoking 14M Python interpreters per second on Blue Waters.
UR - http://www.scopus.com/inward/record.url?scp=84960843731&partnerID=8YFLogxK
U2 - 10.1145/2822332.2822338
DO - 10.1145/2822332.2822338
M3 - Conference contribution
AN - SCOPUS:84960843731
T3 - Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis
BT - Proceedings of WORKS 2015
PB - Association for Computing Machinery, Inc
T2 - 10th Workshop on Workflows in Support of Large-Scale Science, WORKS 2015
Y2 - 15 November 2015
ER -