Distributed Power Control for Two-Tier Femtocell Networks with QoS Provisioning Based on Q-Learning

The explosive growth of mobile multimedia services has caused tremendous network traffic in wireless networks and a great part of the multimedia services are delay-sensitive. Therefore, it is important to design efficient radio resource allocation algorithms to increase network capacity and guarantee the delay QoS. In this paper, we study the power control problem in the downlink of two-tier femtocell networks with the consideration of the delay QoS provisioning. Specifically, we introduce the effective capacity (EC) as the network performance measure instead of the Shannon capacity to provide the statistical delay QoS provisioning. Then, the optimization problem is modeled as a non- cooperative game and the existence of Nash Equilibriums (NE) is investigated. However, in order to enhance the self-organization capacity of femtocells, based on non-cooperative game, we employ a Q-learning framework in which all of the femtocell base stations (FBSs) are considered as agents to achieve power allocation. Then a distributed Q-learning-based power control algorithm is proposed to make femtocell users (FUs) gain maximum EC. Numerical results show that the proposed algorithm can not only maintain the delay requirements of the delay-sensitive services, but also has a good convergence performance.