Abstract
The Oak Ridge Leadership Computing Facility (OLCF) runs the No. 4 supercomputer in the world, supported by a petascale file system, to facilitate scientific discovery. In this paper, using the daily file system metadata snapshots collected over 500 days, we have studied the behavioral trends of 1,362 active users and 380 projects across 35 science domains. In particular, we have analyzed both individual and collective behavior of users and projects, highlighting needs from individual communities and the overall requirements to operate the file system. We have analyzed the metadata across three dimensions, namely (i) the projects' file generation and usage trends, using quantitative file system-centric metrics, (ii) scientific user behavior on the file system, and (iii) the data sharing trends of users and projects. To the best of our knowledge, our work is the first of its kind to provide comprehensive insights on user behavior from multiple science domains through metadata analysis of a large-scale shared file system. We envision that this OLCF case study will provide valuable insights for the design, operation, and management of storage systems at scale, and also encourage other HPC centers to undertake similar such efforts.
Original language | English |
---|---|
Title of host publication | Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017 |
Publisher | Association for Computing Machinery, Inc |
ISBN (Electronic) | 9781450351140 |
DOIs | |
State | Published - Nov 12 2017 |
Event | International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017 - Denver, United States Duration: Nov 12 2017 → Nov 17 2017 |
Publication series
Name | Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017 |
---|
Conference
Conference | International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017 |
---|---|
Country/Territory | United States |
City | Denver |
Period | 11/12/17 → 11/17/17 |
Funding
We would like to thank our shepherd, Neil Chue Hong, for his feedback. The work was supported by, and used the resources of, the Oak Ridge Leadership Computing Facility, located in the National Center for Computational Sciences at ORNL, which is managed by UT Battelle, LLC for the U.S. DOE (under the contract No. DE-AC05-00OR22725).
Keywords
- Distributed file systems
- Usage measurement