Parallel flow in Deep Predictive Coding Networks

This paper proposes a cognitive architecture for sensory processing of multimodal data. The cognitive architectures, referred to as Deep Predictive Coding Networks (DPCN) were first used to model video streams. Here we use DPCNs with two input sources, for example: video and speech recordings. We train DPCNs as generative models of both sensors. Since we constrain the network to have a single hidden code for both inputs, we name the proposed architecture as Multimodal DPCN (MDPCN). Experimental results show that the “parallel” flow between the two sensory modes increases the interclass separability achieved by unsupervised clustering. We validate the proposed method with a multimodal classification task using part of the VIDTIMIT dataset.