Training Reformulated Product Units in Hybrid Neural Networks

Higher order networks allow modelling of correlates and geometrically invariant properties. Current techniques for their development either require domain knowledge, or are constrained by scaling properties or local minima. A novel reformulation of the product unit is introduced, motivated by a desire to improve scaling and training properties. The new unit allows developing high orders of positive and negative powers, and correlates in a single stage, but can be trained successfully using standard back propagation techniques. Tests on standard benchmarks in various hybrid topologies demonstrate the potential in a variety of problem domains.