Word-of-Mouth Understanding: Entity-Centric Multimodal Aspect-Opinion Mining in Social Media

Most existing approaches on aspect-opinion mining focus on the text domain and cannot be applied to social media where the aspects are essentially multimodal and the opinions depend on the specific aspects. To address the problem of multimodal aspect-opinion mining for entities by leveraging multiple cross-collection sources in social media, in this paper we propose a multimodal aspect-opinion model (mmAOM) considering both user-generated photos and textual documents to simultaneously capture correlations between textual and visual modalities, as well as associations between aspects and opinions . By identifying the aspects and the corresponding opinions related to entities, we apply the mmAOM to entity association visualization and multimodal aspect-opinion retrieval. We have conducted extensive experiments on real-world datasets of entities including Flickr photos, Tripadvisor reviews, and news articles. Qualitative and quantitative evaluation results have validated the effectiveness of the multimodal aspect-opinion mining model, and demonstrated the utility of the derived aspects and opinions from mmAOM in applications of entity association visualization and aspect-opinion retrieval.