TY - GEN
T1 - Towards continuous benchmarking
T2 - 6th Platform for Advanced Scientific Computing Conference, PASC 2019
AU - Anzt, Hartwig
AU - Chen, Yen Chen
AU - Cojean, Terry
AU - Dongarra, Jack
AU - Flegar, Goran
AU - Nayak, Pratik
AU - Quintana-Ortí, Enrique S.
AU - Tsai, Yuhsiang M.
AU - Wang, Weichung
N1 - Publisher Copyright:
© 2019 Association for Computing Machinery.
PY - 2019/6/12
Y1 - 2019/6/12
N2 - We present an automated performance evaluation framework that enables an automated workflow for testing and performance evaluation of software libraries. Integrating this component into an ecosystem enables sustainable software development, as a community effort, via a web application for interactively evaluating the performance of individual software components. The performance evaluation tool is based exclusively on web technologies, which removes the burden of downloading performance data or installing additional software. We employ this framework for the Ginkgo software ecosystem, but the framework can be used with essentially any software project, including the comparison between different software libraries. The Continuous Integration (CI) framework of Ginkgo is also extended to automatically run a benchmark suite on predetermined HPC systems, store the state of the machine and the environment along with the compiled binaries, and collect results in a publicly accessible performance data repository based on Git. The Ginkgo performance explorer (GPE) can be used to retrieve the performance data from the repository, and visualizes it in a web browser. GPE also implements an interface that allows users to write scripts, archived in a Git repository, to extract particular data, compute particular metrics, and visualize them in many different formats (as specified by the script). The combination of these approaches creates a workflow which enables performance reproducibility and software sustainability of scientific software. In this paper, we present example scripts that extract and visualize performance data for Ginkgo’s SpMV kernels that allow users to identify the optimal kernel for specific problem characteristics.
AB - We present an automated performance evaluation framework that enables an automated workflow for testing and performance evaluation of software libraries. Integrating this component into an ecosystem enables sustainable software development, as a community effort, via a web application for interactively evaluating the performance of individual software components. The performance evaluation tool is based exclusively on web technologies, which removes the burden of downloading performance data or installing additional software. We employ this framework for the Ginkgo software ecosystem, but the framework can be used with essentially any software project, including the comparison between different software libraries. The Continuous Integration (CI) framework of Ginkgo is also extended to automatically run a benchmark suite on predetermined HPC systems, store the state of the machine and the environment along with the compiled binaries, and collect results in a publicly accessible performance data repository based on Git. The Ginkgo performance explorer (GPE) can be used to retrieve the performance data from the repository, and visualizes it in a web browser. GPE also implements an interface that allows users to write scripts, archived in a Git repository, to extract particular data, compute particular metrics, and visualize them in many different formats (as specified by the script). The combination of these approaches creates a workflow which enables performance reproducibility and software sustainability of scientific software. In this paper, we present example scripts that extract and visualize performance data for Ginkgo’s SpMV kernels that allow users to identify the optimal kernel for specific problem characteristics.
KW - Automated performance benchmarking
KW - Continuous integration
KW - Healthy software lifecycle
KW - Interactive performance visualization
UR - http://www.scopus.com/inward/record.url?scp=85068762806&partnerID=8YFLogxK
U2 - 10.1145/3324989.3325719
DO - 10.1145/3324989.3325719
M3 - Conference contribution
AN - SCOPUS:85068762806
T3 - Proceedings of the Platform for Advanced Scientific Computing Conference, PASC 2019
BT - Proceedings of the Platform for Advanced Scientific Computing Conference, PASC 2019
PB - Association for Computing Machinery, Inc
Y2 - 12 June 2019 through 14 June 2019
ER -
