Bandits with budgets

Motivated by online advertising applications, we consider a version of the classical multi-armed bandit problem where there is a cost associated with pulling each arm, and a corresponding budget which limits the number of times that an arm can be pulled. We derive regret bounds on the expected reward in such a bandit problem using a modification of the well-known upper confidence bound algorithm UCB1.