Analyzing deep learning model inferences for image classification using OpenVINO

Zheming Jin; Hal Finkel

doi:10.1109/IPDPSW50202.2020.00152

Analyzing deep learning model inferences for image classification using OpenVINO

Zheming Jin, Hal Finkel

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

16 Scopus citations

Abstract

It may be desirable to execute deep learning model inferences on an integrated GPU at the edge. While such GPUs are much less powerful than discrete GPUs, it is able to deliver higher floating-point operations per second than a CPU located on the same die. For edge devices, the benefit of moving to lower precision with minimal loss of accuracy to obtain higher performance is also attractive. Hence, we chose 14 deep learning models for image classification to evaluate their inference performance with the OpenVINO toolkit. Then, we analyzed the implementation of the fastest inference model of all the models. The experimental results are promising. Compared to the performance of full-precision (FP32) models, the speedup of the 8-bit (INT8) quantization ranges from 1.02 to 1.56 on an Intel® Xeon® 4-core CPU, and the speedup of the FP16 models ranges from 1.1 to 2 on an Intel® IrisTM Pro GPU. For the FP32 models, the GPU is on average 1.5X faster than the CPU.

Original language	English
Title of host publication	Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	908-911
Number of pages	4
ISBN (Electronic)	9781728174457
DOIs	https://doi.org/10.1109/IPDPSW50202.2020.00152
State	Published - May 2020
Externally published	Yes
Event	34th IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020 - New Orleans, United States Duration: May 18 2020 → May 22 2020

Publication series

Name	Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020

Conference

Conference	34th IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020
Country/Territory	United States
City	New Orleans
Period	05/18/20 → 05/22/20

Funding

ACKNOWLEDGMENT We appreciate the reviewers for their constructive criticism. The research was supported by the U.S. Department of Energy, Office of Science, under contract DE AC0206CH11357 and made use of the Argonne Leadership Computing Facility, a DOE Office of Science User Facility.

Keywords

CPU
Deep learning
GPU
Inference
Quantization

Access to Document

10.1109/IPDPSW50202.2020.00152

Cite this

Jin, Z., & Finkel, H. (2020). Analyzing deep learning model inferences for image classification using OpenVINO. In Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020 (pp. 908-911). Article 9150136 (Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPSW50202.2020.00152

Jin, Zheming ; Finkel, Hal. / Analyzing deep learning model inferences for image classification using OpenVINO. Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020. Institute of Electrical and Electronics Engineers Inc., 2020. pp. 908-911 (Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020).

@inproceedings{e8f2194ca4a94e508c7ac4771bb96bc0,

title = "Analyzing deep learning model inferences for image classification using OpenVINO",

abstract = "It may be desirable to execute deep learning model inferences on an integrated GPU at the edge. While such GPUs are much less powerful than discrete GPUs, it is able to deliver higher floating-point operations per second than a CPU located on the same die. For edge devices, the benefit of moving to lower precision with minimal loss of accuracy to obtain higher performance is also attractive. Hence, we chose 14 deep learning models for image classification to evaluate their inference performance with the OpenVINO toolkit. Then, we analyzed the implementation of the fastest inference model of all the models. The experimental results are promising. Compared to the performance of full-precision (FP32) models, the speedup of the 8-bit (INT8) quantization ranges from 1.02 to 1.56 on an Intel{\textregistered} Xeon{\textregistered} 4-core CPU, and the speedup of the FP16 models ranges from 1.1 to 2 on an Intel{\textregistered} IrisTM Pro GPU. For the FP32 models, the GPU is on average 1.5X faster than the CPU.",

keywords = "CPU, Deep learning, GPU, Inference, Quantization",

author = "Zheming Jin and Hal Finkel",

note = "Publisher Copyright: {\textcopyright} 2020 IEEE.; 34th IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020 ; Conference date: 18-05-2020 Through 22-05-2020",

year = "2020",

month = may,

doi = "10.1109/IPDPSW50202.2020.00152",

language = "English",

series = "Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "908--911",

booktitle = "Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020",

}

Jin, Z & Finkel, H 2020, Analyzing deep learning model inferences for image classification using OpenVINO. in Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020., 9150136, Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020, Institute of Electrical and Electronics Engineers Inc., pp. 908-911, 34th IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020, New Orleans, United States, 05/18/20. https://doi.org/10.1109/IPDPSW50202.2020.00152

Analyzing deep learning model inferences for image classification using OpenVINO. / Jin, Zheming; Finkel, Hal.
Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020. Institute of Electrical and Electronics Engineers Inc., 2020. p. 908-911 9150136 (Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Analyzing deep learning model inferences for image classification using OpenVINO

AU - Jin, Zheming

AU - Finkel, Hal

PY - 2020/5

Y1 - 2020/5

N2 - It may be desirable to execute deep learning model inferences on an integrated GPU at the edge. While such GPUs are much less powerful than discrete GPUs, it is able to deliver higher floating-point operations per second than a CPU located on the same die. For edge devices, the benefit of moving to lower precision with minimal loss of accuracy to obtain higher performance is also attractive. Hence, we chose 14 deep learning models for image classification to evaluate their inference performance with the OpenVINO toolkit. Then, we analyzed the implementation of the fastest inference model of all the models. The experimental results are promising. Compared to the performance of full-precision (FP32) models, the speedup of the 8-bit (INT8) quantization ranges from 1.02 to 1.56 on an Intel® Xeon® 4-core CPU, and the speedup of the FP16 models ranges from 1.1 to 2 on an Intel® IrisTM Pro GPU. For the FP32 models, the GPU is on average 1.5X faster than the CPU.

AB - It may be desirable to execute deep learning model inferences on an integrated GPU at the edge. While such GPUs are much less powerful than discrete GPUs, it is able to deliver higher floating-point operations per second than a CPU located on the same die. For edge devices, the benefit of moving to lower precision with minimal loss of accuracy to obtain higher performance is also attractive. Hence, we chose 14 deep learning models for image classification to evaluate their inference performance with the OpenVINO toolkit. Then, we analyzed the implementation of the fastest inference model of all the models. The experimental results are promising. Compared to the performance of full-precision (FP32) models, the speedup of the 8-bit (INT8) quantization ranges from 1.02 to 1.56 on an Intel® Xeon® 4-core CPU, and the speedup of the FP16 models ranges from 1.1 to 2 on an Intel® IrisTM Pro GPU. For the FP32 models, the GPU is on average 1.5X faster than the CPU.

KW - CPU

KW - Deep learning

KW - GPU

KW - Inference

KW - Quantization

UR - http://www.scopus.com/inward/record.url?scp=85091566559&partnerID=8YFLogxK

U2 - 10.1109/IPDPSW50202.2020.00152

DO - 10.1109/IPDPSW50202.2020.00152

M3 - Conference contribution

AN - SCOPUS:85091566559

T3 - Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020

SP - 908

EP - 911

BT - Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 34th IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020

Y2 - 18 May 2020 through 22 May 2020

ER -

Jin Z, Finkel H. Analyzing deep learning model inferences for image classification using OpenVINO. In Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020. Institute of Electrical and Electronics Engineers Inc. 2020. p. 908-911. 9150136. (Proceedings - 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020). doi: 10.1109/IPDPSW50202.2020.00152

Analyzing deep learning model inferences for image classification using OpenVINO

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this