Using object based files for high performance parallel I/O

Jeremy Logan; Phillip M. Dickens

doi:10.1109/IDAACS.2007.4488394

Using object based files for high performance parallel I/O

Jeremy Logan, Phillip M. Dickens

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Scopus citations

Abstract

We contend that the scalable I/O problem in high performance computing is largely due to the legacy view of a file as a linear sequence of bytes. In this paper we introduce an alternative to the traditional "flat file" that uses the information contained in file views to partition a file into an optimal set of objects, minimizing locking contention and simplifying the lock management strategy. We illustrate the use of an object based cache added to ROMIO to efficiently and transparently add object-based file capabilities to MPI-IO. We analyze the performance of our system using the FLASH-IO benchmark, and demonstrate a substantial performance improvement over the standard ROMIO implementation.

Original language	English
Title of host publication	2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems
Subtitle of host publication	Technology and Applications, IDAACS
Pages	149-154
Number of pages	6
DOIs	https://doi.org/10.1109/IDAACS.2007.4488394
State	Published - 2007
Externally published	Yes
Event	2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS - Dortmund, Germany Duration: Sep 6 2007 → Sep 8 2007

Publication series

Name	2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS

Conference

Conference	2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS
Country/Territory	Germany
City	Dortmund
Period	09/6/07 → 09/8/07

Keywords

File view
I/O performance
MPI-IO
Object-based cache
Object-based files
Parallel I/O

Access to Document

10.1109/IDAACS.2007.4488394

Cite this

Logan, J., & Dickens, P. M. (2007). Using object based files for high performance parallel I/O. In 2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS (pp. 149-154). Article 4488394 (2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS). https://doi.org/10.1109/IDAACS.2007.4488394

@inproceedings{cbb52b3438e44950b9612f396d149cc4,

title = "Using object based files for high performance parallel I/O",

abstract = "We contend that the scalable I/O problem in high performance computing is largely due to the legacy view of a file as a linear sequence of bytes. In this paper we introduce an alternative to the traditional {"}flat file{"} that uses the information contained in file views to partition a file into an optimal set of objects, minimizing locking contention and simplifying the lock management strategy. We illustrate the use of an object based cache added to ROMIO to efficiently and transparently add object-based file capabilities to MPI-IO. We analyze the performance of our system using the FLASH-IO benchmark, and demonstrate a substantial performance improvement over the standard ROMIO implementation.",

keywords = "File view, I/O performance, MPI-IO, Object-based cache, Object-based files, Parallel I/O",

author = "Jeremy Logan and Dickens, {Phillip M.}",

year = "2007",

doi = "10.1109/IDAACS.2007.4488394",

language = "English",

isbn = "1424413486",

series = "2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS",

pages = "149--154",

booktitle = "2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems",

note = "2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS ; Conference date: 06-09-2007 Through 08-09-2007",

}

Logan, J & Dickens, PM 2007, Using object based files for high performance parallel I/O. in 2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS., 4488394, 2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS, pp. 149-154, 2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS, Dortmund, Germany, 09/6/07. https://doi.org/10.1109/IDAACS.2007.4488394

Using object based files for high performance parallel I/O. / Logan, Jeremy; Dickens, Phillip M.
2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS. 2007. p. 149-154 4488394 (2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Using object based files for high performance parallel I/O

AU - Logan, Jeremy

AU - Dickens, Phillip M.

PY - 2007

Y1 - 2007

N2 - We contend that the scalable I/O problem in high performance computing is largely due to the legacy view of a file as a linear sequence of bytes. In this paper we introduce an alternative to the traditional "flat file" that uses the information contained in file views to partition a file into an optimal set of objects, minimizing locking contention and simplifying the lock management strategy. We illustrate the use of an object based cache added to ROMIO to efficiently and transparently add object-based file capabilities to MPI-IO. We analyze the performance of our system using the FLASH-IO benchmark, and demonstrate a substantial performance improvement over the standard ROMIO implementation.

AB - We contend that the scalable I/O problem in high performance computing is largely due to the legacy view of a file as a linear sequence of bytes. In this paper we introduce an alternative to the traditional "flat file" that uses the information contained in file views to partition a file into an optimal set of objects, minimizing locking contention and simplifying the lock management strategy. We illustrate the use of an object based cache added to ROMIO to efficiently and transparently add object-based file capabilities to MPI-IO. We analyze the performance of our system using the FLASH-IO benchmark, and demonstrate a substantial performance improvement over the standard ROMIO implementation.

KW - File view

KW - I/O performance

KW - MPI-IO

KW - Object-based cache

KW - Object-based files

KW - Parallel I/O

UR - http://www.scopus.com/inward/record.url?scp=50149085802&partnerID=8YFLogxK

U2 - 10.1109/IDAACS.2007.4488394

DO - 10.1109/IDAACS.2007.4488394

M3 - Conference contribution

AN - SCOPUS:50149085802

SN - 1424413486

SN - 9781424413485

T3 - 2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS

SP - 149

EP - 154

BT - 2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems

T2 - 2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS

Y2 - 6 September 2007 through 8 September 2007

ER -

Logan J, Dickens PM. Using object based files for high performance parallel I/O. In 2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS. 2007. p. 149-154. 4488394. (2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS). doi: 10.1109/IDAACS.2007.4488394

Using object based files for high performance parallel I/O

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this