Q-learning based supplier-agents for electricity markets

In this paper the problem of designing supplier-agents for electricity markets using reinforcement learning (RL) algorithm is discussed. With the help of these agents we are able to run and simulate competitions among several suppliers of electric energy in forward electricity markets (e.g. day-ahead energy markets). The goal of each supplier-agent (SA) is to maximize its revenue for the entire trading period (e.g. 24 hours in a day-ahead market). We use a temporal difference (TD) method for each SA to basically learn the market environment and the opportunities that give it the maximum revenue. In Q-learning (QL) algorithm, the aim of each SA is to maximize its expected reward function over the entire trading period. Each SA takes its market strategy in each period by either exploring or exploiting. Exploitation is executed by comparing the revenue of a period with the average revenue of the previous periods or the agents´ target revenue. In exploration the agent tries to learn its market environment by trial-error. If exploration gives higher revenue to the agent then it will increase the probability of taking exploratory actions, otherwise the probability of greedy actions will be increased. An IEEE 30-bus system with six supplier-agents is used for market simulation studies.