Optimal management of energy storage system based on reinforcement learning

Energy storage system consists of distributed generation, storage device, loads and some intelligent control devices in the smart grid. It enables energy flow from the storage device to the grid. An amount of balancing energy is procured to meet the load demand when there is a deficit in power generation. The excessive distributed generation power of storage device can either be sold to the grid or be used to provide frequency regulation service. Real-time pricing techniques would greatly influence the system control center in deciding when to sell power, buy power or provide regulation service. The power of distributed generation, load demand, electricity price and the frequency regulation price are independent of each other. Each of the four stochastic processes is modeled as a Markov process to reflect the dynamic characteristics. The optimal control problem of deciding when to sell power, buy power or provide regulation service is formulated as a semi-Markov decision process. The Sarsa algorithm is used to adapt the control operation in order to maximize the long-term rewards on the basis of meeting the load demand. Simulation results show a significant increase of total rewards, a faster convergence speed and good effect with the proposed algorithm.