Fusion of EEG Topograhic Features and fMRI Using Canonical Partial Least Squares

In this paper we present a novel method for describing the EEG as a sequence of topographies, based on the notion of microstates. We use Hidden Markov Models (HMM) to model the temporal evolution of the topography of the average Event Related Potential (ERP) and we calculate the Fisher score of the sequence by taking the gradient of the trained model parameters given the sequence. In this context, the average Event Related Potential (ERP) is described as a sequence of topographies and the Fisher score describes how this sequence deviates from the learned HMM. This alternative modeling of the ERP is used to fuse EEG information, as expressed by the temporal evolution of the topography, and Functional Magnetic Resonance Imaging (fMRI). We use Canonical Partial Least Squares (CPLS) for the fusion of the Fisher score with fMRI features. In order to test the effectiveness of this method, we compare the results of this methodology with the results of CPLS using the average ERP signal of a single channel. Using this methodology we are able to derive components that co-vary between EEG and fMRI and present significant differences between the two tasks. The results indicate that this descriptor effectively characterizes the temporal evolution of the ERP topography and can be used for fusing EEG and fMRI for the discrimination of the brain activity on different tasks.