Face system evaluation toolkit: Recognition is harder than it seems

Challenges for face recognition still exist in factors such as pose, blur and distance. Many current datasets containing mostly frontal images are regarded as being too easy. With obviously unsolved problems researchers are in need of datasets that test these remaining challenges. There are quite a few datasets in existence to study pose. Datasets to study blur and distance are almost non-existent. Datasets allowing for the study of these variables would prove to be useful to researchers in biométrie surveillance applications. However, until now there has been no effective way to create datasets that encompass these three variables in a controlled fashion. Toolsets exist for testing algorithms, but not systems. Designing and creating toolsets to produce a well controlled dataset or to test the full end-to-end recognition system is not trivial. While the use of real subjects may produce the most realistic dataset, it is not always a practical solution and it limits repeatability making the comparison of systems impractical. This paper attempts to address the dataset issue in two ways. The foremost is to introduce a new toolset that allows for the manipulation and capture of synthetic data. With this toolset researchers can not only generate their own datasets, they can do so in real environments to better approximate operational scenarios. Secondly, we provide challenge datasets generated from our validated framework as a first set of data for other researchers. These datasets allow for the study of blur, pose and distance. Overall, this work provides researchers with a new ability to evaluate entire face recognition systems from image acquisition to recognition scores.