Cell Range Expansion Using Distributed Q-Learning in Heterogeneous Networks

Cell range expansion (CRE) is a technique to expand a pico cell range virtually by adding a bias value to the pico received power, instead of increasing transmit power of pico base station (PBS), so that coverage, cell-edge throughput, and overall network throughput are improved. Many studies have focused on inter-cell interference coordination (ICIC) in CRE, because macro base station´s (MBS´s) strong transmit power harms the expanded region (ER) user equipments (UEs) that select PBSs by bias value. Optimal bias value that minimizes the number of UE outages depends on several factors such as the dividing ratio of radio resources between MBSs and PBSs. In addition it varies from UE to another. Thus, most papers use the common bias value among all UEs determined by a trial and error method. In this paper we propose a scheme to determine the bias value of each UE by using Q-learning algorithm where each UE learns its bias value that minimizes the number of UE outages from its past experience independently. Simulation results show that, compared to the scheme using optimal common bias value, the proposed scheme reduces the number of UE outages and improves network throughput.