Traffic Signal Optimization by Integrating Reinforcement Learning and Digital Twins

Vijayalakshmi K. Kumarasamy; Abhilasha Jairam Saroj; Yu Liang; Dalei Wu; Michael P. Hunter; Angshuman Guin; Mina Sartipi

doi:10.1109/SWC57546.2023.10448974

Traffic Signal Optimization by Integrating Reinforcement Learning and Digital Twins

Vijayalakshmi K. Kumarasamy, Abhilasha Jairam Saroj, Yu Liang, Dalei Wu, Michael P. Hunter, Angshuman Guin, Mina Sartipi

Applied Research for Mobility Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Scopus citations

Abstract

Machine learning (ML) methods, especially reinforcement learning (RL), have been widely considered for traffic signal optimization in intelligent transportation systems. Most of these ML methods are centralized, lacking in scalability and adaptability in large traffic networks. Further, it is challenging to train such ML models due to the lack of training platforms and/or the cost of deploying and training in a real traffic networks. This paper presents an approach for the integration of decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin (DT) to optimize traffic signals for the reduction of traffic congestion and network-wide fuel consumption related to stopping. Specifically, the DGMARL agents learn traffic state patterns and make decisions regarding traffic signal control with assistance from a Digital Twin module, which simulates and replicates the traffic behaviors of a real traffic network. The proposed approach was evaluated using PTV-Vissim [1], a microscopic traffic simulation platform. PTV-Vissim is also the simulation engine of the DT, enabling emulation and optimization of the traffic signals on the MLK Smart Corridor in Chattanooga, Tennessee. Compared to an actuated signal control baseline approach, experiment results show that Eco_PI, a developed performance measure capturing the impact of stops on fuel consumption, was reduced by 44.27% in a 24-hour and an average of 29.88% in a PM-peak-hour scenario.

Original language	English
Title of host publication	Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9798350319804
DOIs	https://doi.org/10.1109/SWC57546.2023.10448974
State	Published - 2023
Event	9th IEEE Smart World Congress, SWC 2023 - Portsmouth, United Kingdom Duration: Aug 28 2023 → Aug 31 2023

Publication series

Name	Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023

Conference

Conference	9th IEEE Smart World Congress, SWC 2023
Country/Territory	United Kingdom
City	Portsmouth
Period	08/28/23 → 08/31/23

Funding

This manuscript has been partly authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The publisher acknowledges the US government license to provide public access under the DOE Public Access Plan (https://energy.gov/downloads/doe-public-access-plan). ACKNOWLEDGMENT This material is based upon work supported by the U.S. Department of Energy’s Office of Energy Efficiency and Renewable Energy under the Award Number DE-EE0009208.

Keywords

Digital Twin
Fuel Consumption
Graph Neural Network
Multi-Agent Reinforcement Learning
Traffic Signal Optimization

Access to Document

10.1109/SWC57546.2023.10448974

Cite this

Kumarasamy, V. K., Saroj, A. J., Liang, Y., Wu, D., Hunter, M. P., Guin, A., & Sartipi, M. (2023). Traffic Signal Optimization by Integrating Reinforcement Learning and Digital Twins. In Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023 (Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SWC57546.2023.10448974

Kumarasamy, Vijayalakshmi K. ; Saroj, Abhilasha Jairam ; Liang, Yu et al. / Traffic Signal Optimization by Integrating Reinforcement Learning and Digital Twins. Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023).

@inproceedings{70230b3c9a1341ebb65b5f7c3f2c3915,

title = "Traffic Signal Optimization by Integrating Reinforcement Learning and Digital Twins",

abstract = "Machine learning (ML) methods, especially reinforcement learning (RL), have been widely considered for traffic signal optimization in intelligent transportation systems. Most of these ML methods are centralized, lacking in scalability and adaptability in large traffic networks. Further, it is challenging to train such ML models due to the lack of training platforms and/or the cost of deploying and training in a real traffic networks. This paper presents an approach for the integration of decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin (DT) to optimize traffic signals for the reduction of traffic congestion and network-wide fuel consumption related to stopping. Specifically, the DGMARL agents learn traffic state patterns and make decisions regarding traffic signal control with assistance from a Digital Twin module, which simulates and replicates the traffic behaviors of a real traffic network. The proposed approach was evaluated using PTV-Vissim [1], a microscopic traffic simulation platform. PTV-Vissim is also the simulation engine of the DT, enabling emulation and optimization of the traffic signals on the MLK Smart Corridor in Chattanooga, Tennessee. Compared to an actuated signal control baseline approach, experiment results show that Eco\_PI, a developed performance measure capturing the impact of stops on fuel consumption, was reduced by 44.27\% in a 24-hour and an average of 29.88\% in a PM-peak-hour scenario.",

keywords = "Digital Twin, Fuel Consumption, Graph Neural Network, Multi-Agent Reinforcement Learning, Traffic Signal Optimization",

author = "Kumarasamy, \{Vijayalakshmi K.\} and Saroj, \{Abhilasha Jairam\} and Yu Liang and Dalei Wu and Hunter, \{Michael P.\} and Angshuman Guin and Mina Sartipi",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 9th IEEE Smart World Congress, SWC 2023 ; Conference date: 28-08-2023 Through 31-08-2023",

year = "2023",

doi = "10.1109/SWC57546.2023.10448974",

language = "English",

series = "Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023",

}

Kumarasamy, VK, Saroj, AJ, Liang, Y, Wu, D, Hunter, MP, Guin, A & Sartipi, M 2023, Traffic Signal Optimization by Integrating Reinforcement Learning and Digital Twins. in Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023. Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023, Institute of Electrical and Electronics Engineers Inc., 9th IEEE Smart World Congress, SWC 2023, Portsmouth, United Kingdom, 08/28/23. https://doi.org/10.1109/SWC57546.2023.10448974

Traffic Signal Optimization by Integrating Reinforcement Learning and Digital Twins. / Kumarasamy, Vijayalakshmi K.; Saroj, Abhilasha Jairam; Liang, Yu et al.
Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Traffic Signal Optimization by Integrating Reinforcement Learning and Digital Twins

AU - Kumarasamy, Vijayalakshmi K.

AU - Saroj, Abhilasha Jairam

AU - Liang, Yu

AU - Wu, Dalei

AU - Hunter, Michael P.

AU - Guin, Angshuman

AU - Sartipi, Mina

PY - 2023

Y1 - 2023

N2 - Machine learning (ML) methods, especially reinforcement learning (RL), have been widely considered for traffic signal optimization in intelligent transportation systems. Most of these ML methods are centralized, lacking in scalability and adaptability in large traffic networks. Further, it is challenging to train such ML models due to the lack of training platforms and/or the cost of deploying and training in a real traffic networks. This paper presents an approach for the integration of decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin (DT) to optimize traffic signals for the reduction of traffic congestion and network-wide fuel consumption related to stopping. Specifically, the DGMARL agents learn traffic state patterns and make decisions regarding traffic signal control with assistance from a Digital Twin module, which simulates and replicates the traffic behaviors of a real traffic network. The proposed approach was evaluated using PTV-Vissim [1], a microscopic traffic simulation platform. PTV-Vissim is also the simulation engine of the DT, enabling emulation and optimization of the traffic signals on the MLK Smart Corridor in Chattanooga, Tennessee. Compared to an actuated signal control baseline approach, experiment results show that Eco_PI, a developed performance measure capturing the impact of stops on fuel consumption, was reduced by 44.27% in a 24-hour and an average of 29.88% in a PM-peak-hour scenario.

AB - Machine learning (ML) methods, especially reinforcement learning (RL), have been widely considered for traffic signal optimization in intelligent transportation systems. Most of these ML methods are centralized, lacking in scalability and adaptability in large traffic networks. Further, it is challenging to train such ML models due to the lack of training platforms and/or the cost of deploying and training in a real traffic networks. This paper presents an approach for the integration of decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin (DT) to optimize traffic signals for the reduction of traffic congestion and network-wide fuel consumption related to stopping. Specifically, the DGMARL agents learn traffic state patterns and make decisions regarding traffic signal control with assistance from a Digital Twin module, which simulates and replicates the traffic behaviors of a real traffic network. The proposed approach was evaluated using PTV-Vissim [1], a microscopic traffic simulation platform. PTV-Vissim is also the simulation engine of the DT, enabling emulation and optimization of the traffic signals on the MLK Smart Corridor in Chattanooga, Tennessee. Compared to an actuated signal control baseline approach, experiment results show that Eco_PI, a developed performance measure capturing the impact of stops on fuel consumption, was reduced by 44.27% in a 24-hour and an average of 29.88% in a PM-peak-hour scenario.

KW - Digital Twin

KW - Fuel Consumption

KW - Graph Neural Network

KW - Multi-Agent Reinforcement Learning

KW - Traffic Signal Optimization

UR - https://www.scopus.com/pages/publications/85187371719

U2 - 10.1109/SWC57546.2023.10448974

DO - 10.1109/SWC57546.2023.10448974

M3 - Conference contribution

AN - SCOPUS:85187371719

T3 - Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023

BT - Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 9th IEEE Smart World Congress, SWC 2023

Y2 - 28 August 2023 through 31 August 2023

ER -

Kumarasamy VK, Saroj AJ, Liang Y, Wu D, Hunter MP, Guin A et al. Traffic Signal Optimization by Integrating Reinforcement Learning and Digital Twins. In Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023. Institute of Electrical and Electronics Engineers Inc. 2023. (Proceedings - 2023 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing and Data Security, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PCDS/Metaverse 2023). doi: 10.1109/SWC57546.2023.10448974

Traffic Signal Optimization by Integrating Reinforcement Learning and Digital Twins

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this