Dimensionality effects on the Markov property in Shape Memory Alloy hysteretic environment

Shape memory alloy actuators can be used for morphing, or shape change, by controlling their temperature, which is effectively done by applying a voltage difference across their length. Control of these actuators requires determination of the relationship between voltage and strain so that an input-output map can be developed. To determine this policy and map the hysteretic region, a reinforcement learning algorithm called Sarsa was used. Proper use of reinforcement learning requires that the learning environment have the Markov property. However, hysteresis spaces are commonly referenced as non-Markovian due to the fact that state history is needed to properly predict future states and rewards. This paper reveals that this formerly non-Markovian learning environment of shape memory alloy hysteresis can become Markovian by means of increasing the dimensionality of the measured states. The paper compares learning attempts in both versions of the environment and will show that reinforcement learning is successful in the modified learning environment by learning a near-optimal policy for controlling the length of a shape memory alloy wire. This is then validated by using the modified reinforcement learning agent to learn a near-optimal control policy in an experimental setting.