Beyond face: Improving person clustering in consumer photos by exploring contextual information

Automatic person clustering, which groups photos based on the individuals appearing in a photo collection, is a key component to facilitate photo management and sharing. Traditionally, person clusters are basically built by detecting faces and matching facial features. But these facial clusters can perform poorly when there are large pose variations and occlusions, which are not uncommon in consumer photos. In this paper, we propose an approach that employs contextual information to complement facial information in order to significantly improve the performance of person clustering. The proposed system is able to detect human skin, hair and clothing regions and extract features robustly. By matching the features, high-precision contextual clusters are obtained which can automatically link together multiple face clusters of the same person for efficient annotation. Promising results on family photo collections demonstrated the effectiveness of our approach.