Estimating Biped Gait Using Spline-Based Probability Distribution Function With Q-Learning

This paper studies the probability distribution functions of the parameters to be learned and optimized in biped gait generation. By formulating the gait pattern generation into a multiobjective optimization problem with consideration of geometric and state constraints, dynamically stable and low energy cost biped gaits are generated and optimized by the proposed method, namely Spline-based Estimation of Distribution Algorithm (EDA) with Q-learning updating rule (EDA_S_Q). Instead of assuming variables as independent ones, the relationship between them is exploited by formulating the corresponding probability models with the Catmull-Rom cubic spline function. Such kind of function is proved to be a suboptimal and adaptive realization of the cubic spline function and is capable of providing high-precision description. Moreover, the probability models are updated autonomously by Q-learning method, which is model-free and adaptive. Thus, EDA_S_Q can deal with complex probability distribution functions without a prior knowledge about the distribution. The biped gait generated by EDA_S_Q has been verified using the simulation model of a humanoid soccer robot Robo-Erectus. It also shows that EDA_S_Q can generate the desired biped gaits autonomously in short learning epochs. An interpretation of the transition probability distribution achieved by EDA_S_Q provides us easy understanding for biped locomotion and better control in humanoid robots.