Research Issues in Multiple Policy Optimization Using Collaborative Reinforcement Learning

Self-organizing techniques have successfully been used to optimize software systems, such as optimization of route stability in ad hoc network routing and optimization of the use of storage space or processing power using load balancing. Existing self-organizing techniques typically focus on a single, usually implicitly specified, system goal and tune systems parameters towards optimally meeting that goal. In this paper, we consider optimization of large-scale multi-agent ubiquitous computing environments, such as urban traffic control. Applications in this class are typically required to optimize towards multiple goals simultaneously. Additionally, these multiple goals can potentially be conflicting, change over time, and apply to various parts of the system such as a single agent, a group of agents, or the system as a whole. In contrast to existing self-organizing systems in which agents are homogeneous to the extent that they are working towards a common goal, agents in these systems are heterogeneous in that they may have differing goals. Thus, existing self-organizing optimization techniques must be extended to deal with multiple goal optimization and the resulting heterogeneity of agents. In this paper we present a research agenda for extending collaborative reinforcement learning (CRL), an existing self-organizing optimization technique, to support multiple policy optimization.