Routing based on layered stochastic games for multi-hop cognitive radio networks

This paper proposes a distributed routing mechanism for the multi-hop Cognitive Radio Networks (CRNs) over multiple primary channels. The Secondary Users (SUs) attempts to utilize the channels and minimize their delay along the route while avoiding causing interference to the Primary Users (PUs). In order to address the problem of time-varying channel condition due to the PU dynamics, the route-selection process is modeled as a global Markov Decision Process (MDP). We show that such a global routing MDP can be decomposed into the layered MDPs, in which the interactions between neighbor SUs with their local next-hop selection are modeled as the local stochastic games. By applying reinforcement learning with utility based fictitious play, the best response of each SU can be learned from the local game with the only need for the information exchange from next-hop SUs. The proposed algorithm is evaluated through simulations and is shown to be effective in reducing the delays for multiple flows in the CRN.