Traffic signal control using reinforcement learning with webster method

Abstract

Conventional traffic signal control algorithms are static, linear, fuzzy, or non-deterministic. Therefore, they are inappropriate for dealing with transportation systems efficiently. However, the Webster method is one conventional method that can find the optimal solution in one junction. Model predictive control methods have high computing costs in simulation. Machine-learning algorithms have become available as communication technology, the Internet of Things (IoT), and Big Data have improved. One example is reinforcement learning, which has an agent that continually interacts with the environment and is appropriate for data-driven approaches. However, the research on integrating reinforcement learning with traffic signal control usually uses image or matrix data as input, which results in long time requirements for training and difficulty in achieving convergence. Furthermore, a simplified action space is used, such as by expanding the current signal phase or switching to other phases. One model is also trained for one demand scenario, which lacks robustness. Therefore, in this study, the signal control variables of the green ratio and cycle length are defined as actions using a Double Deep-Q Network (DDQN) or the Webster method, and the robustness is improved for various travel patterns. Although reinforcement learning algorithms can find a proper solution in one junction, a trade-off between intersections is required when there are four junctions, which is not possible with the Webster method alone. The results show that with four junctions, the original DDQN and the DDQN initialized with the Webster method provide the best trade-off between intersections and achieve high network performance, but the discrepancy between the two methods is not large. This study contributes to the concept of combining reinforcement learning with traffic signal control variables and algorithms. Further study is needed to develop the concept by applying other reinforcement algorithms or considering other signal control variables such as the offset.