Thompson Sampling for Opportunistic Spectrum Access with Markovian Rewards

This paper considers the problem of multi- channels opportunistic access in stationary wireless environments, where an intelligent and unlicensed agent uses the existing licensed spectrum to maximize it´s throughput while not causing harm to existing licensed users. Each channel is modeled as unknown and never ending stream of binary data generated by first order discrete time Markov process with two states Gilbert-Elliot model. This problem is classified as restless bandit, where optimal solution is intractable. An on-line approximation algorithm is proposed based on Thompson Sampling Algorithm, which acts as heuristic search for best channel in the spectrum. Empirical evaluations are presented at the end of the paper that show an improved performance compared to other existing algorithms in cases where channels are not bursty.