MDLoader: A Hybrid Model-driven Data Loader for Distributed Deep Neural Networks Training

Jonghyun Bae; Jong Youl Choi; Massimiliano Lupo Pasini; Kshitij Mehta; Khaled Z. Ibrahim

doi:10.1109/IPDPSW63119.2024.00203

MDLoader: A Hybrid Model-driven Data Loader for Distributed Deep Neural Networks Training

Jonghyun Bae, Jong Youl Choi, Massimiliano Lupo Pasini, Kshitij Mehta, Khaled Z. Ibrahim

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

In this work, we propose MD Loader, a hybrid in-memory data loader for distributed deep neural networks. MDLoader introduces a model-driven performance estimator to automatically switch between one-sided and collective communication at runtime.

Original language	English
Title of host publication	2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1193-1195
Number of pages	3
ISBN (Electronic)	9798350364606
DOIs	https://doi.org/10.1109/IPDPSW63119.2024.00203
State	Published - 2024
Event	2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024 - San Francisco, United States Duration: May 27 2024 → May 31 2024

Publication series

Name	2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024

Conference

Conference	2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024
Country/Territory	United States
City	San Francisco
Period	05/27/24 → 05/31/24

Keywords

Collective communication
Graph Neural Network
One-sided communication
Performance estimator

Access to Document

10.1109/IPDPSW63119.2024.00203

Cite this

Bae, J., Choi, J. Y., Pasini, M. L., Mehta, K., & Ibrahim, K. Z. (2024). MDLoader: A Hybrid Model-driven Data Loader for Distributed Deep Neural Networks Training. In 2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024 (pp. 1193-1195). (2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPSW63119.2024.00203

Bae, Jonghyun ; Choi, Jong Youl ; Pasini, Massimiliano Lupo et al. / MDLoader : A Hybrid Model-driven Data Loader for Distributed Deep Neural Networks Training. 2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024. Institute of Electrical and Electronics Engineers Inc., 2024. pp. 1193-1195 (2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024).

@inproceedings{b02b581e66b34d6788bb05f39c0f98b5,

title = "MDLoader: A Hybrid Model-driven Data Loader for Distributed Deep Neural Networks Training",

abstract = "In this work, we propose MD Loader, a hybrid in-memory data loader for distributed deep neural networks. MDLoader introduces a model-driven performance estimator to automatically switch between one-sided and collective communication at runtime.",

keywords = "Collective communication, Graph Neural Network, One-sided communication, Performance estimator",

author = "Jonghyun Bae and Choi, {Jong Youl} and Pasini, {Massimiliano Lupo} and Kshitij Mehta and Ibrahim, {Khaled Z.}",

note = "Publisher Copyright: {\textcopyright} 2024 IEEE.; 2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024 ; Conference date: 27-05-2024 Through 31-05-2024",

year = "2024",

doi = "10.1109/IPDPSW63119.2024.00203",

language = "English",

series = "2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1193--1195",

booktitle = "2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024",

}

Bae, J, Choi, JY , Pasini, ML , Mehta, K & Ibrahim, KZ 2024, MDLoader: A Hybrid Model-driven Data Loader for Distributed Deep Neural Networks Training. in 2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024. 2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024, Institute of Electrical and Electronics Engineers Inc., pp. 1193-1195, 2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024, San Francisco, United States, 05/27/24. https://doi.org/10.1109/IPDPSW63119.2024.00203

MDLoader: A Hybrid Model-driven Data Loader for Distributed Deep Neural Networks Training. / Bae, Jonghyun; Choi, Jong Youl ; Pasini, Massimiliano Lupo et al.
2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024. Institute of Electrical and Electronics Engineers Inc., 2024. p. 1193-1195 (2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - MDLoader

T2 - 2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024

AU - Bae, Jonghyun

AU - Choi, Jong Youl

AU - Pasini, Massimiliano Lupo

AU - Mehta, Kshitij

AU - Ibrahim, Khaled Z.

PY - 2024

Y1 - 2024

N2 - In this work, we propose MD Loader, a hybrid in-memory data loader for distributed deep neural networks. MDLoader introduces a model-driven performance estimator to automatically switch between one-sided and collective communication at runtime.

AB - In this work, we propose MD Loader, a hybrid in-memory data loader for distributed deep neural networks. MDLoader introduces a model-driven performance estimator to automatically switch between one-sided and collective communication at runtime.

KW - Collective communication

KW - Graph Neural Network

KW - One-sided communication

KW - Performance estimator

UR - http://www.scopus.com/inward/record.url?scp=85200775237&partnerID=8YFLogxK

U2 - 10.1109/IPDPSW63119.2024.00203

DO - 10.1109/IPDPSW63119.2024.00203

M3 - Conference contribution

AN - SCOPUS:85200775237

T3 - 2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024

SP - 1193

EP - 1195

BT - 2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 27 May 2024 through 31 May 2024

ER -

Bae J, Choi JY , Pasini ML , Mehta K, Ibrahim KZ. MDLoader: A Hybrid Model-driven Data Loader for Distributed Deep Neural Networks Training. In 2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024. Institute of Electrical and Electronics Engineers Inc. 2024. p. 1193-1195. (2024 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2024). doi: 10.1109/IPDPSW63119.2024.00203

MDLoader: A Hybrid Model-driven Data Loader for Distributed Deep Neural Networks Training

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this