Characterizing a high throughput computing workload: The compact muon solenoid (CMS) experiment at LHC

Rafael Ferreira Da Silva; Mats Rynge; Gideon Juve; Igor Sfiligoi; Ewa Deelman; James Letts; Frank Wurthwein; Miron Livny

doi:10.1016/j.procs.2015.05.190

Characterizing a high throughput computing workload: The compact muon solenoid (CMS) experiment at LHC

Rafael Ferreira Da Silva, Mats Rynge, Gideon Juve, Igor Sfiligoi, Ewa Deelman, James Letts, Frank Wurthwein, Miron Livny

Research output: Contribution to journal › Conference article › peer-review

6 Scopus citations

Abstract

High throughput computing (HTC) has aided the scientific community in the analysis of vast amounts of data and computational jobs in distributed environments. To manage these large workloads, several systems have been developed to efficiently allocate and provide access to distributed resources. Many of these systems rely on job characteristics estimates (e.g., job runtime) to characterize the workload behavior, which in practice is hard to obtain. In this work, we perform an exploratory analysis of the CMS experiment workload using the statistical recursive partitioning method and conditional inference trees to identify patterns that characterize particular behaviors of the workload. We then propose an estimation process to predict job characteristics based on the collected data. Experimental results show that our process estimates job runtime with 75% of accuracy on average, and produces nearly optimal predictions for disk and memory consumption.

Original language	English
Pages (from-to)	39-48
Number of pages	10
Journal	Procedia Computer Science
Volume	51
Issue number	1
DOIs	https://doi.org/10.1016/j.procs.2015.05.190
State	Published - 2015
Externally published	Yes
Event	International Conference on Computational Science, ICCS 2002 - Amsterdam, Netherlands Duration: Apr 21 2002 → Apr 24 2002

Funding

This work was funded by DOE under the contract number ER26110, “dV/dt - Accelerating the Rate of Progress Towards Extreme Scale Collaborative Science”. We also thank Jeff Dost and Greg Thain for their valuable help, and the Open Science Grid (OSG).

Funders	Funder number
U.S. Department of Energy	ER26110
U.S. Department of Energy

Keywords

High throughput computing
Job characteristics estimation
Workload characterization

Access to Document

10.1016/j.procs.2015.05.190

Cite this

@article{ae50ae8f70ae42f794868c4a4b6ed989,

title = "Characterizing a high throughput computing workload: The compact muon solenoid (CMS) experiment at LHC",

abstract = "High throughput computing (HTC) has aided the scientific community in the analysis of vast amounts of data and computational jobs in distributed environments. To manage these large workloads, several systems have been developed to efficiently allocate and provide access to distributed resources. Many of these systems rely on job characteristics estimates (e.g., job runtime) to characterize the workload behavior, which in practice is hard to obtain. In this work, we perform an exploratory analysis of the CMS experiment workload using the statistical recursive partitioning method and conditional inference trees to identify patterns that characterize particular behaviors of the workload. We then propose an estimation process to predict job characteristics based on the collected data. Experimental results show that our process estimates job runtime with 75% of accuracy on average, and produces nearly optimal predictions for disk and memory consumption.",

keywords = "High throughput computing, Job characteristics estimation, Workload characterization",

author = "{Da Silva}, {Rafael Ferreira} and Mats Rynge and Gideon Juve and Igor Sfiligoi and Ewa Deelman and James Letts and Frank Wurthwein and Miron Livny",

note = "Publisher Copyright: {\textcopyright} The Authors. Published by Elsevier B.V.; International Conference on Computational Science, ICCS 2002 ; Conference date: 21-04-2002 Through 24-04-2002",

year = "2015",

doi = "10.1016/j.procs.2015.05.190",

language = "English",

volume = "51",

pages = "39--48",

journal = "Procedia Computer Science",

issn = "1877-0509",

publisher = "Elsevier",

number = "1",

}

TY - JOUR

T1 - Characterizing a high throughput computing workload

T2 - International Conference on Computational Science, ICCS 2002

AU - Da Silva, Rafael Ferreira

AU - Rynge, Mats

AU - Juve, Gideon

AU - Sfiligoi, Igor

AU - Deelman, Ewa

AU - Letts, James

AU - Wurthwein, Frank

AU - Livny, Miron

N1 - Publisher Copyright: © The Authors. Published by Elsevier B.V.

PY - 2015

Y1 - 2015

N2 - High throughput computing (HTC) has aided the scientific community in the analysis of vast amounts of data and computational jobs in distributed environments. To manage these large workloads, several systems have been developed to efficiently allocate and provide access to distributed resources. Many of these systems rely on job characteristics estimates (e.g., job runtime) to characterize the workload behavior, which in practice is hard to obtain. In this work, we perform an exploratory analysis of the CMS experiment workload using the statistical recursive partitioning method and conditional inference trees to identify patterns that characterize particular behaviors of the workload. We then propose an estimation process to predict job characteristics based on the collected data. Experimental results show that our process estimates job runtime with 75% of accuracy on average, and produces nearly optimal predictions for disk and memory consumption.

AB - High throughput computing (HTC) has aided the scientific community in the analysis of vast amounts of data and computational jobs in distributed environments. To manage these large workloads, several systems have been developed to efficiently allocate and provide access to distributed resources. Many of these systems rely on job characteristics estimates (e.g., job runtime) to characterize the workload behavior, which in practice is hard to obtain. In this work, we perform an exploratory analysis of the CMS experiment workload using the statistical recursive partitioning method and conditional inference trees to identify patterns that characterize particular behaviors of the workload. We then propose an estimation process to predict job characteristics based on the collected data. Experimental results show that our process estimates job runtime with 75% of accuracy on average, and produces nearly optimal predictions for disk and memory consumption.

KW - High throughput computing

KW - Job characteristics estimation

KW - Workload characterization

UR - http://www.scopus.com/inward/record.url?scp=84939136672&partnerID=8YFLogxK

U2 - 10.1016/j.procs.2015.05.190

DO - 10.1016/j.procs.2015.05.190

M3 - Conference article

AN - SCOPUS:84939136672

SN - 1877-0509

VL - 51

SP - 39

EP - 48

JO - Procedia Computer Science

JF - Procedia Computer Science

IS - 1

Y2 - 21 April 2002 through 24 April 2002

ER -

Characterizing a high throughput computing workload: The compact muon solenoid (CMS) experiment at LHC

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this