Distributed learning in cognitive radio networks: Multi-armed bandit with distributed multiple players

We consider a cognitive radio network with distributed multiple secondary users, where each user independently searches for spectrum opportunities in multiple channels without exchanging information with others. The occupancy of each channel is modeled as an i.i.d. Bernoulli process with unknown mean. Users choosing the same channel collide, and none or only one receives reward depending on the collision model. This problem can be formulated as a decentralized multi-armed bandit problem. We measure the performance of a decentralized policy by the system regret, defined as the total reward loss with respect to the optimal performance under the perfect scenario where all channel parameters are known to all users and collisions among secondary users are eliminated through perfect scheduling. We show that the minimum system regret grows with time at the same logarithmic order as in the centralized counterpart, where users exchange observations and make decisions jointly. We propose a basic policy structure that ensures a Time Division Fair Sharing (TDFS) of the channels. Based on this basic TDFS structure, decentralized policies can be constructed to achieve this optimal order while ensuring fairness among users. Furthermore, we show that the proposed TDFS policy belongs to a general class of decentralized polices, for which a uniform performance benchmark is established. All results hold for general stochastic processes beyond Bernoulli and thus find a wide area of potential applications including multi-channel communication systems, multi-agent systems, web search and advertising, and social networks.