DocumentCode :
1765837
Title :
Learning Computational Models of Video Memorability from fMRI Brain Imaging
Author :
Junwei Han ; Changyuan Chen ; Ling Shao ; Xintao Hu ; Jungong Han ; Tianming Liu
Author_Institution :
Sch. of Autom., Northwestern Polytech. Univ., Xi´an, China
Volume :
45
Issue :
8
fYear :
2015
fDate :
Aug. 2015
Firstpage :
1692
Lastpage :
1703
Abstract :
Generally, various visual media are unequally memorable by the human brain. This paper looks into a new direction of modeling the memorability of video clips and automatically predicting how memorable they are by learning from brain functional magnetic resonance imaging (fMRI). We propose a novel computational framework by integrating the power of low-level audiovisual features and brain activity decoding via fMRI. Initially, a user study experiment is performed to create a ground truth database for measuring video memorability and a set of effective low-level audiovisual features is examined in this database. Then, human subjects´ brain fMRI data are obtained when they are watching the video clips. The fMRI-derived features that convey the brain activity of memorizing videos are extracted using a universal brain reference system. Finally, due to the fact that fMRI scanning is expensive and time-consuming, a computational model is learned on our benchmark dataset with the objective of maximizing the correlation between the low-level audiovisual features and the fMRI-derived features using joint subspace learning. The learned model can then automatically predict the memorability of videos without fMRI scans. Evaluations on publically available image and video databases demonstrate the effectiveness of the proposed framework.
Keywords :
biomedical MRI; feature extraction; medical image processing; psychology; audiovisual features; brain activity decoding; brain functional magnetic resonance imaging; computational model learning; fMRI brain imaging; joint subspace learning; universal brain reference system; video memorability; Brain models; Computational modeling; Feature extraction; Predictive models; Visualization; Audiovisual features; brain imaging; semantic gap; video memorability (VM);
fLanguage :
English
Journal_Title :
Cybernetics, IEEE Transactions on
Publisher :
ieee
ISSN :
2168-2267
Type :
jour
DOI :
10.1109/TCYB.2014.2358647
Filename :
6919270
Link To Document :
بازگشت