ICDARTS: Improving the Stability of Cyclic DARTS

Emily Herron; Steven R. Young; Derek Rose

doi:10.1109/ICMLA55696.2022.00175

ICDARTS: Improving the Stability of Cyclic DARTS

Emily Herron, Steven R. Young, Derek Rose

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Cyclic DARTS (CDARTS) is a Differentiable Architecture Search (DARTS)-based approach to neural architecture search (NAS) that uses a cyclic feedback mechanism to train search and evaluation networks concurrently. This training protocol aims to optimize the search process and evaluate the deep evaluation network comprised of discretized candidate operations. However, this approach introduces a loss function for the evaluation network dependent on the search network. The dissimilarity between the evaluation network's loss function used during the search and retraining phases results in a search network that is a sub-optimal proxy for the final evaluation network accessed during retraining. We present a revised approach that removes the dependency of the evaluation network weights upon those of the search network. In addition, we introduce a modified process for relaxing the search network's zero operations that allows these operations to be retained in the final evaluation networks.

Original language	English
Title of host publication	Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022
Editors	M. Arif Wani, Mehmed Kantardzic, Vasile Palade, Daniel Neagu, Longzhi Yang, Kit-Yan Chan
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1055-1062
Number of pages	8
ISBN (Electronic)	9781665462839
DOIs	https://doi.org/10.1109/ICMLA55696.2022.00175
State	Published - 2022
Event	21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022 - Nassau, Bahamas Duration: Dec 12 2022 → Dec 14 2022

Publication series

Name	Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022

Conference

Conference	21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022
Country/Territory	Bahamas
City	Nassau
Period	12/12/22 → 12/14/22

Funding

This research used resources of the Compute and Data Environment for Science (CADES) at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. Notice: This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Funders	Funder number
CADES
Data Environment for Science
U.S. Department of Energy	DE-AC05-00OR22725
Office of Science

Keywords

AutoML
Deep Learning
Neural Architecture Search
Neural Networks

Access to Document

10.1109/ICMLA55696.2022.00175

Cite this

Herron, E., Young, S. R., & Rose, D. (2022). ICDARTS: Improving the Stability of Cyclic DARTS. In M. A. Wani, M. Kantardzic, V. Palade, D. Neagu, L. Yang, & K.-Y. Chan (Eds.), Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022 (pp. 1055-1062). (Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICMLA55696.2022.00175

Herron, Emily ; Young, Steven R. ; Rose, Derek. / ICDARTS : Improving the Stability of Cyclic DARTS. Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022. editor / M. Arif Wani ; Mehmed Kantardzic ; Vasile Palade ; Daniel Neagu ; Longzhi Yang ; Kit-Yan Chan. Institute of Electrical and Electronics Engineers Inc., 2022. pp. 1055-1062 (Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022).

@inproceedings{04ed749998894be6a85a782873c9e583,

title = "ICDARTS: Improving the Stability of Cyclic DARTS",

abstract = "Cyclic DARTS (CDARTS) is a Differentiable Architecture Search (DARTS)-based approach to neural architecture search (NAS) that uses a cyclic feedback mechanism to train search and evaluation networks concurrently. This training protocol aims to optimize the search process and evaluate the deep evaluation network comprised of discretized candidate operations. However, this approach introduces a loss function for the evaluation network dependent on the search network. The dissimilarity between the evaluation network's loss function used during the search and retraining phases results in a search network that is a sub-optimal proxy for the final evaluation network accessed during retraining. We present a revised approach that removes the dependency of the evaluation network weights upon those of the search network. In addition, we introduce a modified process for relaxing the search network's zero operations that allows these operations to be retained in the final evaluation networks.",

keywords = "AutoML, Deep Learning, Neural Architecture Search, Neural Networks",

author = "Emily Herron and Young, {Steven R.} and Derek Rose",

note = "Publisher Copyright: {\textcopyright} 2022 IEEE.; 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022 ; Conference date: 12-12-2022 Through 14-12-2022",

year = "2022",

doi = "10.1109/ICMLA55696.2022.00175",

language = "English",

series = "Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1055--1062",

editor = "Wani, {M. Arif} and Mehmed Kantardzic and Vasile Palade and Daniel Neagu and Longzhi Yang and Kit-Yan Chan",

booktitle = "Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022",

}

Herron, E , Young, SR & Rose, D 2022, ICDARTS: Improving the Stability of Cyclic DARTS. in MA Wani, M Kantardzic, V Palade, D Neagu, L Yang & K-Y Chan (eds), Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022. Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022, Institute of Electrical and Electronics Engineers Inc., pp. 1055-1062, 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022, Nassau, Bahamas, 12/12/22. https://doi.org/10.1109/ICMLA55696.2022.00175

ICDARTS: Improving the Stability of Cyclic DARTS. / Herron, Emily ; Young, Steven R.; Rose, Derek.
Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022. ed. / M. Arif Wani; Mehmed Kantardzic; Vasile Palade; Daniel Neagu; Longzhi Yang; Kit-Yan Chan. Institute of Electrical and Electronics Engineers Inc., 2022. p. 1055-1062 (Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - ICDARTS

T2 - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022

AU - Herron, Emily

AU - Young, Steven R.

AU - Rose, Derek

PY - 2022

Y1 - 2022

N2 - Cyclic DARTS (CDARTS) is a Differentiable Architecture Search (DARTS)-based approach to neural architecture search (NAS) that uses a cyclic feedback mechanism to train search and evaluation networks concurrently. This training protocol aims to optimize the search process and evaluate the deep evaluation network comprised of discretized candidate operations. However, this approach introduces a loss function for the evaluation network dependent on the search network. The dissimilarity between the evaluation network's loss function used during the search and retraining phases results in a search network that is a sub-optimal proxy for the final evaluation network accessed during retraining. We present a revised approach that removes the dependency of the evaluation network weights upon those of the search network. In addition, we introduce a modified process for relaxing the search network's zero operations that allows these operations to be retained in the final evaluation networks.

AB - Cyclic DARTS (CDARTS) is a Differentiable Architecture Search (DARTS)-based approach to neural architecture search (NAS) that uses a cyclic feedback mechanism to train search and evaluation networks concurrently. This training protocol aims to optimize the search process and evaluate the deep evaluation network comprised of discretized candidate operations. However, this approach introduces a loss function for the evaluation network dependent on the search network. The dissimilarity between the evaluation network's loss function used during the search and retraining phases results in a search network that is a sub-optimal proxy for the final evaluation network accessed during retraining. We present a revised approach that removes the dependency of the evaluation network weights upon those of the search network. In addition, we introduce a modified process for relaxing the search network's zero operations that allows these operations to be retained in the final evaluation networks.

KW - AutoML

KW - Deep Learning

KW - Neural Architecture Search

KW - Neural Networks

UR - http://www.scopus.com/inward/record.url?scp=85152215271&partnerID=8YFLogxK

U2 - 10.1109/ICMLA55696.2022.00175

DO - 10.1109/ICMLA55696.2022.00175

M3 - Conference contribution

AN - SCOPUS:85152215271

T3 - Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022

SP - 1055

EP - 1062

BT - Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022

A2 - Wani, M. Arif

A2 - Kantardzic, Mehmed

A2 - Palade, Vasile

A2 - Neagu, Daniel

A2 - Yang, Longzhi

A2 - Chan, Kit-Yan

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 12 December 2022 through 14 December 2022

ER -

Herron E , Young SR , Rose D. ICDARTS: Improving the Stability of Cyclic DARTS. In Wani MA, Kantardzic M, Palade V, Neagu D, Yang L, Chan KY, editors, Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022. Institute of Electrical and Electronics Engineers Inc. 2022. p. 1055-1062. (Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022). doi: 10.1109/ICMLA55696.2022.00175

ICDARTS: Improving the Stability of Cyclic DARTS

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this