IBench: A Distributed Inference Simulation and Benchmark Suite

Wesley Brewer; Greg Behm; Alan Scheinine; Ben Parsons; Wesley Emeneker; Robert P. Trevino

doi:10.1109/HPEC43674.2020.9286169

IBench: A Distributed Inference Simulation and Benchmark Suite

Wesley Brewer, Greg Behm, Alan Scheinine, Ben Parsons, Wesley Emeneker, Robert P. Trevino

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Scopus citations

Abstract

We present a novel distributed inference benchmarking system, called 'iBench', that provides relevant performance metrics for high-performance edge computing systems using trained deep learning models. The proposed benchmark is unique in that it includes data transfer performance through a distributed system, such as a supercomputer, using clients and servers to provide a system-level benchmark. iBench is flexible and robust enough to allow for the benchmarking of custom-built inference servers. This was demonstrated through the development of a custom Flask-based inference server to serve MLPerf's official ResNet50v1.5 model. In this paper, we compare iBench against MLPerf inference performance on an 8-V100 GPU node. iBench is shown to provide two primary advantages over MLPerf: (1) the ability to measure distributed inference performance, and (2) a more realistic measure of benchmark performance for inference servers on HPC by taking into account additional factors to inference time, such as HTTP request-response time, payload pre-processing and packing time, and invest time.

Original language	English
Title of host publication	2020 IEEE High Performance Extreme Computing Conference, HPEC 2020
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728192192
DOIs	https://doi.org/10.1109/HPEC43674.2020.9286169
State	Published - Sep 22 2020
Externally published	Yes
Event	2020 IEEE High Performance Extreme Computing Conference, HPEC 2020 - Virtual, Waltham, United States Duration: Sep 21 2020 → Sep 25 2020

Publication series

Name	2020 IEEE High Performance Extreme Computing Conference, HPEC 2020

Conference

Conference	2020 IEEE High Performance Extreme Computing Conference, HPEC 2020
Country/Territory	United States
City	Virtual, Waltham
Period	09/21/20 → 09/25/20

Funding

This material is based upon work supported by, or in part by, the Department of Defense High Performance Computing Modernization Program (HPCMP) under User Productivity, Enhanced Technology Transfer, and Training (PET) contracts #GS04T09DBC0017 and #47QFSA18K0111. Any opinions, finding and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the DoD HPCMP.

Keywords

GPU
ResNet50
TensorRT
benchmark
distributed
inference

Access to Document

10.1109/HPEC43674.2020.9286169

Cite this

Brewer, W., Behm, G., Scheinine, A., Parsons, B., Emeneker, W., & Trevino, R. P. (2020). IBench: A Distributed Inference Simulation and Benchmark Suite. In 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020 Article 9286169 (2020 IEEE High Performance Extreme Computing Conference, HPEC 2020). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/HPEC43674.2020.9286169

@inproceedings{0154c3fd21ab4389bb8cd3883d90b3fa,

title = "IBench: A Distributed Inference Simulation and Benchmark Suite",

abstract = "We present a novel distributed inference benchmarking system, called 'iBench', that provides relevant performance metrics for high-performance edge computing systems using trained deep learning models. The proposed benchmark is unique in that it includes data transfer performance through a distributed system, such as a supercomputer, using clients and servers to provide a system-level benchmark. iBench is flexible and robust enough to allow for the benchmarking of custom-built inference servers. This was demonstrated through the development of a custom Flask-based inference server to serve MLPerf's official ResNet50v1.5 model. In this paper, we compare iBench against MLPerf inference performance on an 8-V100 GPU node. iBench is shown to provide two primary advantages over MLPerf: (1) the ability to measure distributed inference performance, and (2) a more realistic measure of benchmark performance for inference servers on HPC by taking into account additional factors to inference time, such as HTTP request-response time, payload pre-processing and packing time, and invest time.",

keywords = "GPU, ResNet50, TensorRT, benchmark, distributed, inference",

author = "Wesley Brewer and Greg Behm and Alan Scheinine and Ben Parsons and Wesley Emeneker and Trevino, {Robert P.}",

note = "Publisher Copyright: {\textcopyright} 2020 IEEE.; 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020 ; Conference date: 21-09-2020 Through 25-09-2020",

year = "2020",

month = sep,

day = "22",

doi = "10.1109/HPEC43674.2020.9286169",

language = "English",

series = "2020 IEEE High Performance Extreme Computing Conference, HPEC 2020",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2020 IEEE High Performance Extreme Computing Conference, HPEC 2020",

}

Brewer, W, Behm, G, Scheinine, A, Parsons, B, Emeneker, W & Trevino, RP 2020, IBench: A Distributed Inference Simulation and Benchmark Suite. in 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020., 9286169, 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020, Institute of Electrical and Electronics Engineers Inc., 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020, Virtual, Waltham, United States, 09/21/20. https://doi.org/10.1109/HPEC43674.2020.9286169

IBench: A Distributed Inference Simulation and Benchmark Suite. / Brewer, Wesley; Behm, Greg; Scheinine, Alan et al.
2020 IEEE High Performance Extreme Computing Conference, HPEC 2020. Institute of Electrical and Electronics Engineers Inc., 2020. 9286169 (2020 IEEE High Performance Extreme Computing Conference, HPEC 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - IBench

T2 - 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020

AU - Brewer, Wesley

AU - Behm, Greg

AU - Scheinine, Alan

AU - Parsons, Ben

AU - Emeneker, Wesley

AU - Trevino, Robert P.

PY - 2020/9/22

Y1 - 2020/9/22

N2 - We present a novel distributed inference benchmarking system, called 'iBench', that provides relevant performance metrics for high-performance edge computing systems using trained deep learning models. The proposed benchmark is unique in that it includes data transfer performance through a distributed system, such as a supercomputer, using clients and servers to provide a system-level benchmark. iBench is flexible and robust enough to allow for the benchmarking of custom-built inference servers. This was demonstrated through the development of a custom Flask-based inference server to serve MLPerf's official ResNet50v1.5 model. In this paper, we compare iBench against MLPerf inference performance on an 8-V100 GPU node. iBench is shown to provide two primary advantages over MLPerf: (1) the ability to measure distributed inference performance, and (2) a more realistic measure of benchmark performance for inference servers on HPC by taking into account additional factors to inference time, such as HTTP request-response time, payload pre-processing and packing time, and invest time.

AB - We present a novel distributed inference benchmarking system, called 'iBench', that provides relevant performance metrics for high-performance edge computing systems using trained deep learning models. The proposed benchmark is unique in that it includes data transfer performance through a distributed system, such as a supercomputer, using clients and servers to provide a system-level benchmark. iBench is flexible and robust enough to allow for the benchmarking of custom-built inference servers. This was demonstrated through the development of a custom Flask-based inference server to serve MLPerf's official ResNet50v1.5 model. In this paper, we compare iBench against MLPerf inference performance on an 8-V100 GPU node. iBench is shown to provide two primary advantages over MLPerf: (1) the ability to measure distributed inference performance, and (2) a more realistic measure of benchmark performance for inference servers on HPC by taking into account additional factors to inference time, such as HTTP request-response time, payload pre-processing and packing time, and invest time.

KW - GPU

KW - ResNet50

KW - TensorRT

KW - benchmark

KW - distributed

KW - inference

UR - http://www.scopus.com/inward/record.url?scp=85099378855&partnerID=8YFLogxK

U2 - 10.1109/HPEC43674.2020.9286169

DO - 10.1109/HPEC43674.2020.9286169

M3 - Conference contribution

AN - SCOPUS:85099378855

T3 - 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020

BT - 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 21 September 2020 through 25 September 2020

ER -

Brewer W, Behm G, Scheinine A, Parsons B, Emeneker W, Trevino RP. IBench: A Distributed Inference Simulation and Benchmark Suite. In 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020. Institute of Electrical and Electronics Engineers Inc. 2020. 9286169. (2020 IEEE High Performance Extreme Computing Conference, HPEC 2020). doi: 10.1109/HPEC43674.2020.9286169

IBench: A Distributed Inference Simulation and Benchmark Suite

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this