Application-level regression testing framework using Jenkins

Reuben Budiardja; Timothy Bouvet; Galen Arnold

doi:10.1002/cpe.4339

Application-level regression testing framework using Jenkins

Reuben Budiardja, Timothy Bouvet, Galen Arnold

Advanced Computing for Nuclear, Particle

Research output: Contribution to journal › Article › peer-review

3 Scopus citations

Abstract

Monitoring and testing for regression of large-scale systems such as the NCSA's Blue Waters supercomputer are challenging tasks. In this paper, we describe the solution we came up with to perform those tasks. Our goal was to find an automated solution for running user-level regression tests to evaluate system usability and performance. Jenkins, an automation server software, was chosen for its versatility, large user base, and multitude of plugins including collecting data and plotting test results over time. We describe our Jenkins deployment to launch and monitor jobs on remote HPC system, perform authentication with one-time password, and integrate with our LDAP server for its authorization. We show some use cases and describe our best practices for successfully using Jenkins as a user-level system-wide regression testing and monitoring framework for large supercomputer systems.

Original language	English
Article number	e4339
Journal	Concurrency and Computation: Practice and Experience
Volume	30
Issue number	1
DOIs	https://doi.org/10.1002/cpe.4339
State	Published - Jan 10 2018

Funding

This material is based upon work performed using computational resources at the National Institute for Computational Sciences (NICS). NICS is supported by the University of Tennessee and Oak Ridge National Laboratory through the Joint Institute for Computational Sciences.¶ This material is part of the Blue Waters sustained-petascale computing project, which is supported by the National Science Foundation (awards OCI-0725070 and ACI-1238993) and the state of Illinois. Blue Waters is a joint effort of the University of Illinois at Urbana-Champaign and its National Center for Supercomputing Applications. A portion of this work used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the US Department of Energy under Contract No. DE-AC05-00OR22725.

Keywords

applications
benchmarking
performance
regression testing
system-monitoring

Access to Document

10.1002/cpe.4339

Cite this

@article{0423b168682d489da33855130ed57d3a,

title = "Application-level regression testing framework using Jenkins",

abstract = "Monitoring and testing for regression of large-scale systems such as the NCSA's Blue Waters supercomputer are challenging tasks. In this paper, we describe the solution we came up with to perform those tasks. Our goal was to find an automated solution for running user-level regression tests to evaluate system usability and performance. Jenkins, an automation server software, was chosen for its versatility, large user base, and multitude of plugins including collecting data and plotting test results over time. We describe our Jenkins deployment to launch and monitor jobs on remote HPC system, perform authentication with one-time password, and integrate with our LDAP server for its authorization. We show some use cases and describe our best practices for successfully using Jenkins as a user-level system-wide regression testing and monitoring framework for large supercomputer systems.",

keywords = "applications, benchmarking, performance, regression testing, system-monitoring",

author = "Reuben Budiardja and Timothy Bouvet and Galen Arnold",

note = "Publisher Copyright: Copyright {\textcopyright} 2017 John Wiley & Sons, Ltd.",

year = "2018",

month = jan,

day = "10",

doi = "10.1002/cpe.4339",

language = "English",

volume = "30",

journal = "Concurrency and Computation: Practice and Experience",

issn = "1532-0626",

publisher = "wiley",

number = "1",

}

TY - JOUR

T1 - Application-level regression testing framework using Jenkins

AU - Budiardja, Reuben

AU - Bouvet, Timothy

AU - Arnold, Galen

PY - 2018/1/10

Y1 - 2018/1/10

N2 - Monitoring and testing for regression of large-scale systems such as the NCSA's Blue Waters supercomputer are challenging tasks. In this paper, we describe the solution we came up with to perform those tasks. Our goal was to find an automated solution for running user-level regression tests to evaluate system usability and performance. Jenkins, an automation server software, was chosen for its versatility, large user base, and multitude of plugins including collecting data and plotting test results over time. We describe our Jenkins deployment to launch and monitor jobs on remote HPC system, perform authentication with one-time password, and integrate with our LDAP server for its authorization. We show some use cases and describe our best practices for successfully using Jenkins as a user-level system-wide regression testing and monitoring framework for large supercomputer systems.

AB - Monitoring and testing for regression of large-scale systems such as the NCSA's Blue Waters supercomputer are challenging tasks. In this paper, we describe the solution we came up with to perform those tasks. Our goal was to find an automated solution for running user-level regression tests to evaluate system usability and performance. Jenkins, an automation server software, was chosen for its versatility, large user base, and multitude of plugins including collecting data and plotting test results over time. We describe our Jenkins deployment to launch and monitor jobs on remote HPC system, perform authentication with one-time password, and integrate with our LDAP server for its authorization. We show some use cases and describe our best practices for successfully using Jenkins as a user-level system-wide regression testing and monitoring framework for large supercomputer systems.

KW - applications

KW - benchmarking

KW - performance

KW - regression testing

KW - system-monitoring

UR - http://www.scopus.com/inward/record.url?scp=85037168425&partnerID=8YFLogxK

U2 - 10.1002/cpe.4339

DO - 10.1002/cpe.4339

M3 - Article

AN - SCOPUS:85037168425

SN - 1532-0626

VL - 30

JO - Concurrency and Computation: Practice and Experience

JF - Concurrency and Computation: Practice and Experience

IS - 1

M1 - e4339

ER -

Application-level regression testing framework using Jenkins

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this