Conjectural Variation-Based Bidding Strategies with Q-Learning in Electricity Markets

A novel conjectural variation-based bidding strategy combined with a Q-learning algorithm is presented in this paper. Generation companies are modeled as adaptive agents in the electricity markets. Conjectural variation is introduced to model rivals´ reaction to one generation company´s bidding strategy. Q-learning is used to model the bidding behavior of generation companies that can learn and adjust their strategies over time. SA-Q-learning algorithm with Metropolis criterion is applied to balance exploitation and exploration in the reinforcement learning process. A series of stage games represented by short-term market clearings form a repeated game for generation companies. Interactions among market participants can be studied accordingly by the proposed agent-based simulation approach. Case studies demonstrate the learning process and corresponding equilibriums.