Abstract
The analysis of vast amounts of data and the processing of complex computational jobs have traditionally relied upon high performance computing (HPC) systems, which offer reliable and efficient management of large-scale computational and data resources. Understanding these analyses' needs is paramount for designing solutions that can lead to better science, and similarly, understanding the characteristics of the user behavior on those systems is important for improving user experiences on HPC systems. A common approach to gathering data about user behavior is to extract workload characteristics from system log data available only to system administrators. Recently at Oak Ridge Leadership Computing Facility (OLCF), however, we unveiled user behavior about the Summit supercomputer by collecting data from a user's point of view with ordinary Unix commands.In this paper, we discuss the process, challenges, and lessons learned while preparing this dataset for publication and submission to an open data challenge. The original dataset contains personal identifiable information (PII) about the users of OLCF which needed be masked prior to publication, and we determined that anonymization, which scrubs PII completely, destroyed too much of the structure of the data to be interesting for the data challenge. We instead chose to pseudonymize the dataset, which reduced the linkability of the dataset to the users' identities. Pseudonymization is significantly more computationally expensive than anonymization, and the size of our dataset, which is approximately 175 million lines of raw text, necessitated the development of a parallelized workflow that could be reused on different HPC machines. We demonstrate the scaling behavior of the workflow on two leadership class HPC systems at OLCF, and we show that we were able to bring the overall makespan time from an impractical 20+ hours on a single node down to around 2 hours. As a result of this work, we release the entire pseudonymized dataset and make the workflows and source code publicly available.
Original language | English |
---|---|
Title of host publication | Proceedings - 2022 IEEE International Conference on Big Data, Big Data 2022 |
Editors | Shusaku Tsumoto, Yukio Ohsawa, Lei Chen, Dirk Van den Poel, Xiaohua Hu, Yoichi Motomura, Takuya Takagi, Lingfei Wu, Ying Xie, Akihiro Abe, Vijay Raghavan |
Publisher | Institute of Electrical and Electronics Engineers Inc. |
Pages | 3432-3440 |
Number of pages | 9 |
ISBN (Electronic) | 9781665480451 |
DOIs | |
State | Published - 2022 |
Event | 2022 IEEE International Conference on Big Data, Big Data 2022 - Osaka, Japan Duration: Dec 17 2022 → Dec 20 2022 |
Publication series
Name | Proceedings - 2022 IEEE International Conference on Big Data, Big Data 2022 |
---|
Conference
Conference | 2022 IEEE International Conference on Big Data, Big Data 2022 |
---|---|
Country/Territory | Japan |
City | Osaka |
Period | 12/17/22 → 12/20/22 |
Funding
This manuscript has been authored by UT-Battelle, LLC, under contract DEAC05- 00OR22725 with the US Department of Energy (DOE). The publisher acknowledges the US government license to provide public access under the DOE Public Access Plan (http://energy.gov/downloads/doe-public-accessplan). This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The publisher acknowledges the US government license to provide public access under the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725.
Keywords
- Big Data
- High Performance Computing
- Personal Identifiable Information
- pseudonymization
- workflows