Emergent proximo-distal maturation through adaptive exploration

Life-long robot learning in the high-dimensional real world requires guided and structured exploration mechanisms. In this developmental context, we investigate here the use of the recently proposed PI_CMAES² episodic reinforcement learning algorithm, which is able to learn high-dimensional motor tasks through adaptive control of exploration. By studying PI_CMAES² in a reaching task on a simulated arm, we observe two developmental properties. First, we show how PI_CMAES² autonomously and continuously tunes the global exploration/exploitation tradeoff, allowing it to re-adapt to changing tasks. Second, we show how PI_CMAES² spontaneously self-organizes a maturational structure whilst exploring the degrees-of-freedom (DOFs) of the motor space. In particular, it automatically demonstrates the so-called proximo-distal maturation observed in humans: after first freezing distal DOFs while exploring predominantly the most proximal DOF, it progressively frees exploration in DOFs along the proximo-distal body axis. These emergent properties suggest the use of PI_CMAES² as a general tool for studying reinforcement learning of skills in life-long developmental learning contexts.