One Step Beyond Myopic Probing Policy: A Heuristic Lookahead Policy for Multi-Channel Opportunistic Access

In this paper, we consider the probing order and stopping problem arising from the identification of spectrum holes in multi-channel cognitive radio networks, in which a secondary user (SU) seeks to maximize the probability of finding an available channel while minimizing the related probing cost within a long time horizon. This problem can be casted into a restless multi-armed bandit problem, which is proved to be PSPACE-hard. The key point of this problem is the trade-off between exploitation, in which the SU stops probing once an available channel is identified, and exploration, in which the SU continues to probe new channels even after identifying an available channel in order to learn the system state to reduce probing cost in the future. To strike a desirable balance between the two conflicting objectives, we develop a heuristic channel probing policy, termed the v-step lookahead policy, in which the SU makes its decision based on the prediction of system state within the future v steps, with v being a tunable parameter. We conduct an analytical study on the structure of the proposed v-step lookahead policy and demonstrate how the policy can be implemented with linear complexity with respect to the number of channels in the system via a detailed analysis on the 1-step lookahead policy. Numerical experiments between the v-step lookahead policy and myopic probing policy on two representative network scenarios demonstrate the effectiveness of the proposed v-step lookahead policy.