The Mertens Unrolled Network (MU-Net): A high dynamic range fusion neural network for through the windshield driver recognition

Max Ruby; David S. Bolme; Joel Brogan; David Cornett; Baldemar Delgado; Gavin Jager; Christi Johnson; Jose Martinez-Mendoza; Hector Santos-Villalobos; Nisha Srinivas

doi:10.1117/12.2566765

The Mertens Unrolled Network (MU-Net): A high dynamic range fusion neural network for through the windshield driver recognition

Max Ruby, David S. Bolme, Joel Brogan, David Cornett, Baldemar Delgado, Gavin Jager, Christi Johnson, Jose Martinez-Mendoza, Hector Santos-Villalobos, Nisha Srinivas

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Face recognition of vehicle occupants through windshields in unconstrained environments poses a number of unique challenges ranging from glare, poor illumination, driver pose and motion blur. In this paper, we further develop the hardware and software components of a custom vehicle imaging system to better overcome these challenges. After the build out of a physical prototype system that performs High Dynamic Range (HDR) imaging, we collect a small dataset of through-windshield image captures of known drivers. We then reformulate the classical Mertens-Kautz-Van Reeth HDR fusion algorithm as a pre-initialized neural network, which we name the Mertens Unrolled Network (MU-Net), for the purpose of fine-tuning the HDR output of through-windshield images. Reconstructed faces from this novel HDR method are then evaluated and compared against other traditional and experimental HDR methods in a pre-trained state-of-the-art (SOTA) facial recognition pipeline, verifying the efficacy of our approach.

Original language	English
Title of host publication	Autonomous Systems
Subtitle of host publication	Sensors, Processing, and Security for Vehicles and Infrastructure 2020
Editors	Michael C. Dudzik, Stephen M. Jameson
Publisher	SPIE
ISBN (Electronic)	9781510636071
DOIs	https://doi.org/10.1117/12.2566765
State	Published - 2020
Event	Autonomous Systems: Sensors, Processing, and Security for Vehicles and Infrastructure 2020 - Virtual, Online, United States Duration: Apr 27 2020 → May 8 2020

Publication series

Name	Proceedings of SPIE - The International Society for Optical Engineering
Volume	11415
ISSN (Print)	0277-786X
ISSN (Electronic)	1996-756X

Conference

Conference	Autonomous Systems: Sensors, Processing, and Security for Vehicles and Infrastructure 2020
Country/Territory	United States
City	Virtual, Online
Period	04/27/20 → 05/8/20

Funding

Further author information: (Send correspondence to Hector Santos-Villalobos) Hector Santos-Villalobos: E-mail: [email protected], Telephone: 1 865 574 0215 David S. Bolme: E-mail: [email protected], Telephone: 1 865 576 0300 Notice: This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). This research was supported in part by an appointment to the Oak Ridge National Laboratory Post-Bachelors Research Associate Program, sponsored by the U.S. Department of Energy and administered by the Oak Ridge Institute for Science and Education, the U.S. Department of Energy, Office of Science, Office of Workforce Development for Teachers and Scientists (WDTS) under the Science Undergraduate Laboratory Internships Program (SULI), and the Texas A&M University-Kingsville, Office of University Programs, Science and Technology Directorate, Department of Homeland Security Grant Award # 2012-ST -062-000054.

Funders	Funder number
Office of University Programs, Science and Technology Directorate
Office of Workforce Development for Teachers
U.S. Department of Energy
U.S. Department of Homeland Security	2012-ST -062-000054
Office of Science
Workforce Development for Teachers and Scientists
Oak Ridge Institute for Science and Education
Texas A and M University-Kingsville

Keywords

Facial Recognition
HDR
MU-Net
Through Windshield

Access to Document

10.1117/12.2566765

Cite this

Ruby, M., Bolme, D. S., Brogan, J., Cornett, D., Delgado, B., Jager, G., Johnson, C., Martinez-Mendoza, J., Santos-Villalobos, H., & Srinivas, N. (2020). The Mertens Unrolled Network (MU-Net): A high dynamic range fusion neural network for through the windshield driver recognition. In M. C. Dudzik, & S. M. Jameson (Eds.), Autonomous Systems: Sensors, Processing, and Security for Vehicles and Infrastructure 2020 Article 114150B (Proceedings of SPIE - The International Society for Optical Engineering; Vol. 11415). SPIE. https://doi.org/10.1117/12.2566765

Ruby, Max ; Bolme, David S. ; Brogan, Joel et al. / The Mertens Unrolled Network (MU-Net) : A high dynamic range fusion neural network for through the windshield driver recognition. Autonomous Systems: Sensors, Processing, and Security for Vehicles and Infrastructure 2020. editor / Michael C. Dudzik ; Stephen M. Jameson. SPIE, 2020. (Proceedings of SPIE - The International Society for Optical Engineering).

@inproceedings{fa48ac435bea478bb846796db456fec1,

title = "The Mertens Unrolled Network (MU-Net): A high dynamic range fusion neural network for through the windshield driver recognition",

abstract = "Face recognition of vehicle occupants through windshields in unconstrained environments poses a number of unique challenges ranging from glare, poor illumination, driver pose and motion blur. In this paper, we further develop the hardware and software components of a custom vehicle imaging system to better overcome these challenges. After the build out of a physical prototype system that performs High Dynamic Range (HDR) imaging, we collect a small dataset of through-windshield image captures of known drivers. We then reformulate the classical Mertens-Kautz-Van Reeth HDR fusion algorithm as a pre-initialized neural network, which we name the Mertens Unrolled Network (MU-Net), for the purpose of fine-tuning the HDR output of through-windshield images. Reconstructed faces from this novel HDR method are then evaluated and compared against other traditional and experimental HDR methods in a pre-trained state-of-the-art (SOTA) facial recognition pipeline, verifying the efficacy of our approach.",

keywords = "Facial Recognition, HDR, MU-Net, Through Windshield",

author = "Max Ruby and Bolme, {David S.} and Joel Brogan and David Cornett and Baldemar Delgado and Gavin Jager and Christi Johnson and Jose Martinez-Mendoza and Hector Santos-Villalobos and Nisha Srinivas",

note = "Publisher Copyright: {\textcopyright} 2020 SPIE.; Autonomous Systems: Sensors, Processing, and Security for Vehicles and Infrastructure 2020 ; Conference date: 27-04-2020 Through 08-05-2020",

year = "2020",

doi = "10.1117/12.2566765",

language = "English",

series = "Proceedings of SPIE - The International Society for Optical Engineering",

publisher = "SPIE",

editor = "Dudzik, {Michael C.} and Jameson, {Stephen M.}",

booktitle = "Autonomous Systems",

}

Ruby, M, Bolme, DS , Brogan, J , Cornett, D, Delgado, B, Jager, G, Johnson, C, Martinez-Mendoza, J, Santos-Villalobos, H & Srinivas, N 2020, The Mertens Unrolled Network (MU-Net): A high dynamic range fusion neural network for through the windshield driver recognition. in MC Dudzik & SM Jameson (eds), Autonomous Systems: Sensors, Processing, and Security for Vehicles and Infrastructure 2020., 114150B, Proceedings of SPIE - The International Society for Optical Engineering, vol. 11415, SPIE, Autonomous Systems: Sensors, Processing, and Security for Vehicles and Infrastructure 2020, Virtual, Online, United States, 04/27/20. https://doi.org/10.1117/12.2566765

The Mertens Unrolled Network (MU-Net): A high dynamic range fusion neural network for through the windshield driver recognition. / Ruby, Max; Bolme, David S.; Brogan, Joel et al.
Autonomous Systems: Sensors, Processing, and Security for Vehicles and Infrastructure 2020. ed. / Michael C. Dudzik; Stephen M. Jameson. SPIE, 2020. 114150B (Proceedings of SPIE - The International Society for Optical Engineering; Vol. 11415).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - The Mertens Unrolled Network (MU-Net)

T2 - Autonomous Systems: Sensors, Processing, and Security for Vehicles and Infrastructure 2020

AU - Ruby, Max

AU - Bolme, David S.

AU - Brogan, Joel

AU - Cornett, David

AU - Delgado, Baldemar

AU - Jager, Gavin

AU - Johnson, Christi

AU - Martinez-Mendoza, Jose

AU - Santos-Villalobos, Hector

AU - Srinivas, Nisha

PY - 2020

Y1 - 2020

N2 - Face recognition of vehicle occupants through windshields in unconstrained environments poses a number of unique challenges ranging from glare, poor illumination, driver pose and motion blur. In this paper, we further develop the hardware and software components of a custom vehicle imaging system to better overcome these challenges. After the build out of a physical prototype system that performs High Dynamic Range (HDR) imaging, we collect a small dataset of through-windshield image captures of known drivers. We then reformulate the classical Mertens-Kautz-Van Reeth HDR fusion algorithm as a pre-initialized neural network, which we name the Mertens Unrolled Network (MU-Net), for the purpose of fine-tuning the HDR output of through-windshield images. Reconstructed faces from this novel HDR method are then evaluated and compared against other traditional and experimental HDR methods in a pre-trained state-of-the-art (SOTA) facial recognition pipeline, verifying the efficacy of our approach.

AB - Face recognition of vehicle occupants through windshields in unconstrained environments poses a number of unique challenges ranging from glare, poor illumination, driver pose and motion blur. In this paper, we further develop the hardware and software components of a custom vehicle imaging system to better overcome these challenges. After the build out of a physical prototype system that performs High Dynamic Range (HDR) imaging, we collect a small dataset of through-windshield image captures of known drivers. We then reformulate the classical Mertens-Kautz-Van Reeth HDR fusion algorithm as a pre-initialized neural network, which we name the Mertens Unrolled Network (MU-Net), for the purpose of fine-tuning the HDR output of through-windshield images. Reconstructed faces from this novel HDR method are then evaluated and compared against other traditional and experimental HDR methods in a pre-trained state-of-the-art (SOTA) facial recognition pipeline, verifying the efficacy of our approach.

KW - Facial Recognition

KW - HDR

KW - MU-Net

KW - Through Windshield

UR - http://www.scopus.com/inward/record.url?scp=85089064406&partnerID=8YFLogxK

U2 - 10.1117/12.2566765

DO - 10.1117/12.2566765

M3 - Conference contribution

AN - SCOPUS:85089064406

T3 - Proceedings of SPIE - The International Society for Optical Engineering

BT - Autonomous Systems

A2 - Dudzik, Michael C.

A2 - Jameson, Stephen M.

PB - SPIE

Y2 - 27 April 2020 through 8 May 2020

ER -

Ruby M, Bolme DS , Brogan J , Cornett D, Delgado B, Jager G et al. The Mertens Unrolled Network (MU-Net): A high dynamic range fusion neural network for through the windshield driver recognition. In Dudzik MC, Jameson SM, editors, Autonomous Systems: Sensors, Processing, and Security for Vehicles and Infrastructure 2020. SPIE. 2020. 114150B. (Proceedings of SPIE - The International Society for Optical Engineering). doi: 10.1117/12.2566765

The Mertens Unrolled Network (MU-Net): A high dynamic range fusion neural network for through the windshield driver recognition

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this