DocumentCode :
14803
Title :
Automatic Caption Generation for News Images
Author :
Yansong Feng ; Lapata, M.
Author_Institution :
Inst. of Comput. Sci. & Technol., Peking Univ., Beijing, China
Volume :
35
Issue :
4
fYear :
2013
fDate :
Apr-13
Firstpage :
797
Lastpage :
812
Abstract :
This paper is concerned with the task of automatically generating captions for images, which is important for many image-related applications. Examples include video and image retrieval as well as the development of tools that aid visually impaired individuals to access pictorial information. Our approach leverages the vast resource of pictures available on the web and the fact that many of them are captioned and colocated with thematically related documents. Our model learns to create captions from a database of news articles, the pictures embedded in them, and their captions, and consists of two stages. Content selection identifies what the image and accompanying article are about, whereas surface realization determines how to verbalize the chosen content. We approximate content selection with a probabilistic image annotation model that suggests keywords for an image. The model postulates that images and their textual descriptions are generated by a shared set of latent variables (topics) and is trained on a weakly labeled dataset (which treats the captions and associated news articles as image labels). Inspired by recent work in summarization, we propose extractive and abstractive surface realization models. Experimental results show that it is viable to generate captions that are pertinent to the specific content of an image and its associated article, while permitting creativity in the description. Indeed, the output of our abstractive model compares favorably to handwritten captions and is often superior to extractive methods.
Keywords :
Internet; handicapped aids; image processing; image retrieval; probability; vision defects; abstractive surface realization models; automatic caption generation; content selection; extractive surface realization models; image retrieval; image-related applications; news article database; news images; pictorial information access; probabilistic image annotation model; surface realization; textual descriptions; thematically related documents; video retrieval; visually impaired individual aid; Data models; Databases; Humans; Noise measurement; Probabilistic logic; Visualization; Vocabulary; Caption generation; image annotation; summarization; topic models; Databases, Factual; Image Processing, Computer-Assisted; Information Storage and Retrieval; Models, Theoretical; Natural Language Processing; Newspapers; Reproducibility of Results;
fLanguage :
English
Journal_Title :
Pattern Analysis and Machine Intelligence, IEEE Transactions on
Publisher :
ieee
ISSN :
0162-8828
Type :
jour
DOI :
10.1109/TPAMI.2012.118
Filename :
6205758
Link To Document :
بازگشت