Multi-Agent Reinforcement Learning Resources Allocation Method Using Dueling Double Deep Q-Network in Vehicular Networks

IEEE Transactions on Vehicular Technology, vol. 72, pp. 13447–13460

Abstract

The communications between vehicle-to-vehicle (V2V) with high frequency, group sending, group receiving and periodic lead to serious collision of wireless resources and limited system capacity, and the rapid channel changes in high mobility vehicular environments preclude the possibility of collecting accurate instantaneous channel state information at the base station for centralized resource management. For the Internet of Vehicles (IoV), it is a fundamental challenge to achieve low latency and high reliability communication for real-time data interaction over short distances in a complex wireless propagation environment, as well as to attenuate and avoid inter-vehicle interference in the region through a reasonable spectrum allocation. To solve the above problems, this paper proposes a resource allocation (RA) method using dueling double deep Q-network reinforcement learning (RL) with low-dimensional fingerprints and soft-update architecture (D3QN-LS) while constructing a multi-agent model based on a Manhattan grid layout urban virtual environment, with communication links between V2V links acting as agents to reuse vehicle-to-infrastructure (V2I) spectrum resources. In addition, we extend the amount of transmitted data in our work, while adding scenarios where spectrum resources are relatively scarce, i.e. the number of V2V links is significantly larger than the amount of spectrum, to compensate for some of the shortcomings in existing literature studies. We demonstrate that the proposed D3QN-LS algorithm leads to a further improvement in the total capacity of V2I links and the success rate of periodic secure message transmission in V2V links.

Authors 7

1. Yuxin Ji

   Nanjing University of Posts and Telecommunications

   Affiliation as printed

   College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, China

2. Yu Wang

   Nanjing University of Posts and Telecommunications

   Affiliation as printed

   College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, China

3. Haitao Zhao

   Nanjing University of Posts and Telecommunications

   Affiliation as printed

   College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, China

4. Guan Gui

   Nanjing University of Posts and Telecommunications

   Affiliation as printed

   College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, China

5. Haris Gacanin Aachen Institute for Communication Technologies and Embedded Systems

   RWTH Aachen University

   Affiliation as printed

   Institute for Communication Technologies and Embedded Systems, RWTH Aachen University, Aachen, Germany

6. Hikmet Sari

   Nanjing University of Posts and Telecommunications

   Affiliation as printed

   College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, China

7. Fumiyuki Adachi

   Tohoku University

   Affiliation as printed

   International Research Institute of Disaster Science (IRIDeS), Tohoku University, Sendai, Japan