Modeling design and control problems involving neural network surrogates

Dominic Yang, Prasanna Balaprakash, Sven Leyffer

Research output: Contribution to journalArticlepeer-review

5 Scopus citations

Abstract

We consider nonlinear optimization problems that involve surrogate models represented by neural networks. We demonstrate first how to directly embed neural network evaluation into optimization models, highlight a difficulty with this approach that can prevent convergence, and then characterize stationarity of such models. We then present two alternative formulations of these problems in the specific case of feedforward neural networks with ReLU activation: as a mixed-integer optimization problem and as a mathematical program with complementarity constraints. For the latter formulation we prove that stationarity at a point for this problem corresponds to stationarity of the embedded formulation. Each of these formulations may be solved with state-of-the-art optimization methods, and we show how to obtain good initial feasible solutions for these methods. We compare our formulations on three practical applications arising in the design and control of combustion engines, in the generation of adversarial attacks on classifier networks, and in the determination of optimal flows in an oil well network.

Original languageEnglish
Pages (from-to)759-800
Number of pages42
JournalComputational Optimization and Applications
Volume83
Issue number3
DOIs
StatePublished - Dec 2022
Externally publishedYes

Funding

The submitted manuscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory (“Argonne”). Argonne, a U.S. Department of Energy Office of Science laboratory, is operated under Contract No. DE-AC02-06CH11357. The U.S. Government retains for itself, and others acting on its behalf, a paid-up nonexclusive, irrevocable worldwide license in said article to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, by or on behalf of the Government. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan. http://energy.gov/downloads/doe-public-access-plan . This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under Contract DE-AC02-06CH11357. This work was also supported by the U.S. Department of Energy through grant DE-FG02-05ER25694. The first author was also supported through an NSF-MSGI fellowship.

FundersFunder number
NSF-MSGI
U.S. Department of Energy
Office of Science
Advanced Scientific Computing ResearchDE-AC02-06CH11357, DE-FG02-05ER25694

    Keywords

    • Complementarity constraints
    • Machine learning
    • Mixed-integer programming
    • Neural networks
    • Nonlinear programming

    Fingerprint

    Dive into the research topics of 'Modeling design and control problems involving neural network surrogates'. Together they form a unique fingerprint.

    Cite this