Communication-Efficient Orchestrations for URLLC Service via Hierarchical Reinforcement Learning

IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), pp. 1–6

Abstract

Ultra-reliable low latency communications (URLLC) service is envisioned to enable use cases with strict reliability and latency requirements in 5G. One approach for enabling URLLC services is to leverage Reinforcement Learning (RL) to efficiently allocate wireless resources. However, with conventional RL methods, the decision variables (though being deployed at various network layers) are typically optimized in the same control loop, leading to significant practical limitations on the control loop’s delay as well as excessive signaling and energy consumption. In this paper, we propose a multi-agent Hierarchical RL (HRL) framework that enables the implementation of multi-level policies with different control loop timescales. Agents with faster control loops are deployed closer to the base station, while the ones with slower control loops are at the edge or closer to the core network providing high-level guidelines for low-level actions. On a use case from the prior art, with our HRL framework, we optimized the maximum number of retransmissions and transmission power of industrial devices. Our extensive simulation results on the factory automation scenario show that the HRL framework achieves better performance as the baseline single-agent RL method, with significantly less overhead of signal transmissions and delay compared to the one-agent RL methods.

Authors 3

1. Milad Ganjalizadeh

   KTH Royal Institute of Technology · Ericsson (Sweden)

   Affiliation as printed

   Ericsson Research,Sweden

   Ericsson Research, Sweden

   School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden

2. Hossein Shokri‐Ghadikolaei

   Ericsson (Sweden)

   Affiliation as printed

   Ericsson Research,Sweden

   Ericsson Research, Sweden

3. Marina Petrova Aachen Mobile Communications and Computing (MCC)

   RWTH Aachen University · KTH Royal Institute of Technology

   Affiliation as printed

   KTH Royal Institute of Technology,School of Electrical Engineering and Computer Science,Stockholm,Sweden

   Mobile Communications and Computing, RWTH Aachen University, Germany

   School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden