Title :
On the Combinatorial Multi-Armed Bandit Problem with Markovian Rewards
Author :
Gai, Yi ; Krishnamachari, Bhaskar ; Liu, Mingyan
Author_Institution :
Ming Hsieh Dept. of Electr. Eng., Univ. of Southern California, Los Angeles, CA, USA
Abstract :
We consider a combinatorial generalization of the classical multi-armed bandit problem that is defined as follows. There is a given bipartite graph of M users and N≥M resources. For each user-resource pair (i,j), there is an associated state that evolves as an aperiodic irreducible finite-state Markov chain with unknown parameters, with transitions occurring each time the particular user i is allocated resource j. The user i receives a reward that depends on the corresponding state each time it is allocated the resource j. The system objective is to learn the best matching of users to resources so that the long-term sum of the rewards received by all users is maximized. This corresponds to minimizing regret, defined here as the gap between the expected total reward that can be obtained by the best-possible static matching and the expected total reward that can be achieved by a given algorithm. We present a polynomial-storage and polynomial-complexity-per-step matching-learning algorithm for this problem. We show that this algorithm can achieve a regret that is uniformly arbitrarily close to logarithmic in time and polynomial in the number of users and resources. This formulation is broadly applicable to scheduling and switching problems in communication networks including cognitive radio networks and significantly extends prior results in the area.
Keywords :
Markov processes; cognitive radio; computational complexity; graph theory; learning (artificial intelligence); resource allocation; scheduling; telecommunication computing; telecommunication switching; Markovian rewards; aperiodic irreducible finite-state Markov chain; best-possible static matching; bipartite graph; classical multiarmed bandit problem; cognitive radio networks; combinatorial generalization; combinatorial multiarmed bandit problem; communication networks; long-term sum; polynomial-complexity-per-step matching-learning algorithm; polynomial-storage; resource allocation; scheduling; switching problems; system objective; unknown parameters; user-resource pair; Algorithm design and analysis; Cognitive radio; Eigenvalues and eigenfunctions; IEEE Communications Society; Markov processes; Polynomials; Upper bound;
Conference_Titel :
Global Telecommunications Conference (GLOBECOM 2011), 2011 IEEE
Conference_Location :
Houston, TX, USA
Print_ISBN :
978-1-4244-9266-4
Electronic_ISBN :
1930-529X
DOI :
10.1109/GLOCOM.2011.6134244