DORA: Distributed Cognitive Random Access of Unslotted Markovian Channels under Tight Collision Constraints

We consider the design of distributed strategies that allow multiple secondary users to opportunistically access multiple unslotted Markovian channels with unknown parameters and tight collision constraints, a challenging problem setting that has not been well addressed by existing work. An optimal strategy would strike a balance among exploration, which is to measure all the channels to identify the best one(s), exploitation, which is to stay on the currently best channel(s) as much as possible, and competition, that is to spread out users in order to avoid overcrowding the best channel(s). Moreover, a strategy has to abide collision constraint of each channel to become an acceptable one. We first assume known channel parameters and formulate a CNLP (constrained nonlinear programming) problem, which we solved through an algorithm we called DORA-Known that computes an optimal randomized access strategy. Next, We address the online channel-parameter learning problem by transforming it into a problem of DTMC (discrete-time Markov chain) estimation with incomplete data, and solving it with an EM (expectation-maximization) based algorithm. We then propose an algorithm called DORA-Learning that extends DORA-Known to incorporate the online channel learning. The proposed algorithms are evaluated and compared with a state-of-art approach that assumes known channel parameters, and two reinforcement learning based schemes. Experimental results illustrate significant performance gain of the two DORA algorithms over the other three approaches.