Abstract
While a large number of deep learning networks have been studied and published that produce outstanding results on natural image datasets, these datasets only make up a fraction of those to which deep learning can be applied. These datasets include text data, audio data, and arrays of sensors that have very different characteristics than natural images. As these “best” networks for natural images have been largely discovered through experimentation and cannot be proven optimal on some theoretical basis, there is no reason to believe that they are the optimal network for these drastically different datasets. Hyperparameter search is thus often a very important process when applying deep learning to a new problem. In this work we present an evolutionary approach to searching the possible space of network hyperparameters and construction that can scale to 18, 000 nodes. This approach is applied to datasets of varying types and characteristics where we demonstrate the ability to rapidly find best hyperparameters in order to enable practitioners to quickly iterate between idea and result.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of MLHPC 2017 |
| Subtitle of host publication | Machine Learning in HPC Environments - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis |
| Publisher | Association for Computing Machinery, Inc |
| ISBN (Electronic) | 9781450351379 |
| DOIs | |
| State | Published - Nov 12 2017 |
| Event | 2017 Machine Learning in HPC Environments, MLHPC 2017 - Denver, United States Duration: Nov 12 2017 → Nov 17 2017 |
Publication series
| Name | Proceedings of MLHPC 2017: Machine Learning in HPC Environments - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis |
|---|
Conference
| Conference | 2017 Machine Learning in HPC Environments, MLHPC 2017 |
|---|---|
| Country/Territory | United States |
| City | Denver |
| Period | 11/12/17 → 11/17/17 |
Funding
Support for participating scientists was provided by NSF and DOE (USA) by CAPES and CNPq (Brazil), by CoNaCyT (Mexico), by Proyecto Basal FB 0821, CONICYT PIA ACT1413, Fondecyt 3170845 and 11130133 (Chile), by CONCYTEC, DGI-PUCP and IDI/IGI-UNI (Peru), by Latin American Center for Physics (CLAF) and by RAS and the Russian Ministry of Education and Science (Russia). MINERvA is supported using the resources of the Fermi National Accelerator Laboratory (Fermilab), a U.S. Department of Energy, Office of Science, HEP User Facility. Fermilab is managed by Fermi Research Alliance, LLC (FRA), acting under Contract No. DE-AC02-07CH11359, which included the MINERvA construction project. Construction support also was granted by the United States National Science Foundation under Award PHY-0619727 and by the University of Rochester. Notice: This manuscript has been authored by UT-Battelle, LLC under contract DE-AC05-00OR22725, and Fermi Research Alliance, LLC (FRA) under contract DE-AC02-07CH11359 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http: //energy.gov/downloads/doe-public-access-plan). Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only. MLHPC’17, November 12–17, 2017, Denver, CO, USA © 2017 Copyright held by the owner/author(s). Publication rights licensed to Association for Computing Machinery. ACM ISBN 978-1-4503-5137-9/17/11...$15.00 https://doi.org/10.1145/3146347.3146355 Research sponsored by the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory, managed by UT-Battelle, LLC, for the U. S. Department of Energy. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. We would like to thank the MINERvA collaboration for the use of their simulated data and for many useful and stimulating conversations. MINERvA is supported using the resources of the Fermi National Accelerator Laboratory (Fermilab), a U.S. Department of Energy, Office of Science, HEP User Facility. Fermilab is managed by Fermi Research Alliance, LLC (FRA), acting under Contract No. DE-AC02-07CH11359, which included the MINERvA construction project. Construction support also was granted by the United States National Science Foundation under Award PHY-0619727 and by the University of Rochester. Support for participating scientists was provided by NSF and DOE (USA) by CAPES and CNPq (Brazil), by CoNaCyT (Mexico), by Proyecto Basal FB 0821, CONICYT PIA ACT1413, Fondecyt 3170845 and 11130133 (Chile), by CONCYTEC, DGI-PUCP and IDI/IGI-UNI (Peru), by Latin American Center for Physics (CLAF) and by RAS and the Russian Ministry of Education and Science (Russia). This work benefited from the use of the SasView application, originally developed under NSF award DMR-0520547. SasView contains code developed with funding from the European Union’s Horizon 2020 research and innovation programme under the SINE2020 project, grant agreement No. 654000. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. Research sponsored by the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory, managed by UT-Battelle, LLC, for the U. S. Department of Energy. This work benefited from the use of the SasView application, originally developed under NSF award DMR-0520547. SasView contains code developed with funding from the European Union’s Horizon 2020 research and innovation programme under the SINE2020 project, grant agreement No. 654000.
| Funders | Funder number |
|---|---|
| CLAF | |
| DGI-PUCP | |
| Fermi Research Alliance, LLC | |
| IDI | |
| IGI-UNI | |
| Latin American Center for Physics | |
| Proyecto Basal | FB 0821 |
| National Science Foundation | DMR-0520547, PHY-0619727 |
| U.S. Department of Energy | DE-AC05-00OR22725, DE-AC02-07CH11359 |
| ACT1413 | |
| Office of Science | |
| Oak Ridge National Laboratory | |
| Fermilab | |
| University of Rochester | |
| Horizon 2020 Framework Programme | 654000 |
| Rochester Academy of Science | |
| United States-Israel Binational Science Foundation | |
| Coordenação de Aperfeiçoamento de Pessoal de Nível Superior | |
| Agencia Nacional de Investigación y Desarrollo | PIA ACT1413 |
| Fondo Nacional de Desarrollo Científico y Tecnológico | 3170845, 11130133 |
| Consejo Nacional de Ciencia y Tecnología | |
| Ministry of Education and Science of the Russian Federation | |
| Conselho Nacional de Desenvolvimento Científico e Tecnológico | |
| National Science Foundation | |
| Consejo Nacional de Ciencia, Tecnología e Innovación Tecnológica |
Keywords
- Deep learning
- Evolutionary algorithms
- High performance computing
- Hyperparameter optimization