Parallel IO support for meta-computing applications: MPI_Connect IO applied to PACX-MPI

Graham E. Fagg; Edgar Gabriel; Michael Resch; Jack J. Dongarra

doi:10.1007/3-540-45417-9_22

Parallel IO support for meta-computing applications: MPI_Connect IO applied to PACX-MPI

Graham E. Fagg, Edgar Gabriel, Michael Resch, Jack J. Dongarra

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Scopus citations

Abstract

Parallel IO (PIO) support for larger scale computing is becoming more important as application developers better understand its importance in reducing overall execution time by avoiding IO overheads. This situation has been made more critical as processor speed and overall system size has increased at a far greater rate than sequential IO performance. Systems such as MPI_Connect and PACX-MPI allow multiple MPPs to be interconnected, complicating IO issues further. MPI_Connect implemented Parallel IO support for distributed applications in the MPI_Conn_IO package by transferring complete sections of files to remote machines, supporting the case that all the applications and the file storage were completely distributed. This system had a number of performance drawbacks compared to the more common usage of metacomputing where some files and applications have an affinity to a home site and thus less data transfer is required. Here we present the new PACX-MPI PIO system based initially on MPI_Connect IO, and attempt to demonstrate multiple methods of handling MPI PIO that cover a greater number of possible usage scenarios. Given are some preliminary performance results as well as a comparison to other PIO grid systems such as the Asian Pacific GridFarm, and Globus gridFTP, GASS and RIO.

Original language	English
Title of host publication	Recent Advances in Parallel Virtual Machine and Message Passing Interface - 8th European PVM/MPI Users’ Group Meeting, Proceedings
Editors	Yiannis Cotronis, Jack Dongarra
Publisher	Springer Verlag
Pages	135-147
Number of pages	13
ISBN (Print)	3540426094, 9783540426097
DOIs	https://doi.org/10.1007/3-540-45417-9_22
State	Published - 2001
Externally published	Yes
Event	8th European on Parallel Virtual Machine and Message Passing Interface Users’ Group Meeting, 2001 - Santorini/Thera, Greece Duration: Sep 23 2001 → Sep 26 2001

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	2131
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	8th European on Parallel Virtual Machine and Message Passing Interface Users’ Group Meeting, 2001
Country/Territory	Greece
City	Santorini/Thera
Period	09/23/01 → 09/26/01

Access to Document

10.1007/3-540-45417-9_22

Cite this

Fagg, G. E., Gabriel, E., Resch, M., & Dongarra, J. J. (2001). Parallel IO support for meta-computing applications: MPI_Connect IO applied to PACX-MPI. In Y. Cotronis, & J. Dongarra (Eds.), Recent Advances in Parallel Virtual Machine and Message Passing Interface - 8th European PVM/MPI Users’ Group Meeting, Proceedings (pp. 135-147). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 2131). Springer Verlag. https://doi.org/10.1007/3-540-45417-9_22

Fagg, Graham E. ; Gabriel, Edgar ; Resch, Michael et al. / Parallel IO support for meta-computing applications : MPI_Connect IO applied to PACX-MPI. Recent Advances in Parallel Virtual Machine and Message Passing Interface - 8th European PVM/MPI Users’ Group Meeting, Proceedings. editor / Yiannis Cotronis ; Jack Dongarra. Springer Verlag, 2001. pp. 135-147 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{5803e470bb484c69a95ae957022bca1c,

title = "Parallel IO support for meta-computing applications: MPI_Connect IO applied to PACX-MPI",

abstract = "Parallel IO (PIO) support for larger scale computing is becoming more important as application developers better understand its importance in reducing overall execution time by avoiding IO overheads. This situation has been made more critical as processor speed and overall system size has increased at a far greater rate than sequential IO performance. Systems such as MPI_Connect and PACX-MPI allow multiple MPPs to be interconnected, complicating IO issues further. MPI_Connect implemented Parallel IO support for distributed applications in the MPI_Conn_IO package by transferring complete sections of files to remote machines, supporting the case that all the applications and the file storage were completely distributed. This system had a number of performance drawbacks compared to the more common usage of metacomputing where some files and applications have an affinity to a home site and thus less data transfer is required. Here we present the new PACX-MPI PIO system based initially on MPI_Connect IO, and attempt to demonstrate multiple methods of handling MPI PIO that cover a greater number of possible usage scenarios. Given are some preliminary performance results as well as a comparison to other PIO grid systems such as the Asian Pacific GridFarm, and Globus gridFTP, GASS and RIO.",

author = "Fagg, {Graham E.} and Edgar Gabriel and Michael Resch and Dongarra, {Jack J.}",

note = "Publisher Copyright: {\textcopyright} Springer-Verlag Berlin Heidelberg 2001.; 8th European on Parallel Virtual Machine and Message Passing Interface Users{\textquoteright} Group Meeting, 2001 ; Conference date: 23-09-2001 Through 26-09-2001",

year = "2001",

doi = "10.1007/3-540-45417-9_22",

language = "English",

isbn = "3540426094",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "135--147",

editor = "Yiannis Cotronis and Jack Dongarra",

booktitle = "Recent Advances in Parallel Virtual Machine and Message Passing Interface - 8th European PVM/MPI Users{\textquoteright} Group Meeting, Proceedings",

}

Fagg, GE, Gabriel, E, Resch, M & Dongarra, JJ 2001, Parallel IO support for meta-computing applications: MPI_Connect IO applied to PACX-MPI. in Y Cotronis & J Dongarra (eds), Recent Advances in Parallel Virtual Machine and Message Passing Interface - 8th European PVM/MPI Users’ Group Meeting, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 2131, Springer Verlag, pp. 135-147, 8th European on Parallel Virtual Machine and Message Passing Interface Users’ Group Meeting, 2001, Santorini/Thera, Greece, 09/23/01. https://doi.org/10.1007/3-540-45417-9_22

Parallel IO support for meta-computing applications: MPI_Connect IO applied to PACX-MPI. / Fagg, Graham E.; Gabriel, Edgar; Resch, Michael et al.
Recent Advances in Parallel Virtual Machine and Message Passing Interface - 8th European PVM/MPI Users’ Group Meeting, Proceedings. ed. / Yiannis Cotronis; Jack Dongarra. Springer Verlag, 2001. p. 135-147 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 2131).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Parallel IO support for meta-computing applications

T2 - 8th European on Parallel Virtual Machine and Message Passing Interface Users’ Group Meeting, 2001

AU - Fagg, Graham E.

AU - Gabriel, Edgar

AU - Resch, Michael

AU - Dongarra, Jack J.

N1 - Publisher Copyright: © Springer-Verlag Berlin Heidelberg 2001.

PY - 2001

Y1 - 2001

N2 - Parallel IO (PIO) support for larger scale computing is becoming more important as application developers better understand its importance in reducing overall execution time by avoiding IO overheads. This situation has been made more critical as processor speed and overall system size has increased at a far greater rate than sequential IO performance. Systems such as MPI_Connect and PACX-MPI allow multiple MPPs to be interconnected, complicating IO issues further. MPI_Connect implemented Parallel IO support for distributed applications in the MPI_Conn_IO package by transferring complete sections of files to remote machines, supporting the case that all the applications and the file storage were completely distributed. This system had a number of performance drawbacks compared to the more common usage of metacomputing where some files and applications have an affinity to a home site and thus less data transfer is required. Here we present the new PACX-MPI PIO system based initially on MPI_Connect IO, and attempt to demonstrate multiple methods of handling MPI PIO that cover a greater number of possible usage scenarios. Given are some preliminary performance results as well as a comparison to other PIO grid systems such as the Asian Pacific GridFarm, and Globus gridFTP, GASS and RIO.

AB - Parallel IO (PIO) support for larger scale computing is becoming more important as application developers better understand its importance in reducing overall execution time by avoiding IO overheads. This situation has been made more critical as processor speed and overall system size has increased at a far greater rate than sequential IO performance. Systems such as MPI_Connect and PACX-MPI allow multiple MPPs to be interconnected, complicating IO issues further. MPI_Connect implemented Parallel IO support for distributed applications in the MPI_Conn_IO package by transferring complete sections of files to remote machines, supporting the case that all the applications and the file storage were completely distributed. This system had a number of performance drawbacks compared to the more common usage of metacomputing where some files and applications have an affinity to a home site and thus less data transfer is required. Here we present the new PACX-MPI PIO system based initially on MPI_Connect IO, and attempt to demonstrate multiple methods of handling MPI PIO that cover a greater number of possible usage scenarios. Given are some preliminary performance results as well as a comparison to other PIO grid systems such as the Asian Pacific GridFarm, and Globus gridFTP, GASS and RIO.

UR - http://www.scopus.com/inward/record.url?scp=0344403336&partnerID=8YFLogxK

U2 - 10.1007/3-540-45417-9_22

DO - 10.1007/3-540-45417-9_22

M3 - Conference contribution

AN - SCOPUS:0344403336

SN - 3540426094

SN - 9783540426097

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 135

EP - 147

BT - Recent Advances in Parallel Virtual Machine and Message Passing Interface - 8th European PVM/MPI Users’ Group Meeting, Proceedings

A2 - Cotronis, Yiannis

A2 - Dongarra, Jack

PB - Springer Verlag

Y2 - 23 September 2001 through 26 September 2001

ER -

Fagg GE, Gabriel E, Resch M, Dongarra JJ. Parallel IO support for meta-computing applications: MPI_Connect IO applied to PACX-MPI. In Cotronis Y, Dongarra J, editors, Recent Advances in Parallel Virtual Machine and Message Passing Interface - 8th European PVM/MPI Users’ Group Meeting, Proceedings. Springer Verlag. 2001. p. 135-147. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/3-540-45417-9_22

Parallel IO support for meta-computing applications: MPI_Connect IO applied to PACX-MPI

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this