Abstract
There have been many recent advancements in imitation and reinforcement learning for autonomous driving, but existing metrics generally lack the means to capture a wide range of driving behaviors and compare the severity of different failure cases. To address this shortcoming, we introduce Quan-titative Evaluation for Driving (QED), which assesses different aspects of driving behavior including the ability to stay in the center of the lane, avoid weaving and erratic behavior, follow the speed limit, and avoid collisions. We compare scores generated by QED against scores assigned by human evaluators on 30 different drivers and 6 different towns in the CARLA driving simulator. In "easy"evaluation scenarios where better drivers are easily distinguished from worse drivers, QED attains 0.96 Pearson correlation and 0.97 Spearman correlation with human evaluators, similar to the baseline inter-human-evaluator 0.96 Pearson correlation and 0.95 Spearman correlation. In "hard"evaluation scenarios where ranking drivers is more ambiguous, QED attains 0.84 Pearson correlation and 0.74 Spearman correlation with human evaluators, slighter higher than the baseline inter-human-evaluator 0.78 Pearson correlation and 0.7 Spearman correlation. While QED may not capture every characteristic that defines good driving, we consider it an important foundation for reproducibility and standardization in the community.
| Original language | English |
|---|---|
| Title of host publication | 2021 IEEE Intelligent Vehicles Symposium Workshops, IV Workshops 2021 |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| Pages | 257-263 |
| Number of pages | 7 |
| ISBN (Electronic) | 9781665479219 |
| DOIs | |
| State | Published - 2021 |
| Event | 32nd IEEE Intelligent Vehicles Symposium Workshops, IV Workshops 2021 - Nagoya, Japan Duration: Jul 11 2021 → Jul 17 2021 |
Publication series
| Name | IEEE Intelligent Vehicles Symposium, Proceedings |
|---|
Conference
| Conference | 32nd IEEE Intelligent Vehicles Symposium Workshops, IV Workshops 2021 |
|---|---|
| Country/Territory | Japan |
| City | Nagoya |
| Period | 07/11/21 → 07/17/21 |
Funding
This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of the manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-accessplan). This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. This research used resources of the Compute and Data Environment for Science (CADES) at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. This work was presented at the From Benchmarking Behavior Prediction to Socially Compatible Behavior Generation in Autonomous Driving Workshop (WS17), IV2021. YThese authors contributed equally to this work *Oak Ridge National Laboratory, Oak Ridge, TN, 37830, {gaos,paulissensr,colettima,pattonrm}@ornl.gov This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of the manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).