Competitive Live Evaluations of Activity-Recognition Systems

Ensuring the validity and usability of activity recognition approaches requires agreement on a set of standard evaluation methods. Due to the diversity of the sensors and other hardware employed, however, designing, implementing, and accepting standard tests is a difficult task. This article presents an initiative to evaluate activity recognition systems: a living-lab evaluation established through the annual Evaluating Ambient Assisted Living Systems through Competitive Benchmarking-Activity Recognition (EvAAL-AR) competition. In the EvAAL-AR, each team brings its own activity-recognition system; all systems are evaluated live on the same activity scenario performed by an actor. The evaluation criteria attempt to capture practical usability: recognition accuracy, user acceptance, recognition delay, installation complexity, and interoperability with ambient assisted living systems. Here, the authors discuss the competition and the competing systems, focusing on the system that achieved the best recognition accuracy, and the system that was evaluated as the best overall. The authors also discuss lessons learned from the competition and ideas for future development of the competition and of the activity recognition field in general.