Information fusion in human-robot collaboration using neural network representation

In this paper, an algorithm for hard and soft data fusion is developed for tracking moving objects using hard data from sensors on autonomous agents and soft data from human observations. Two main challenges are identified and addressed in this paper: 1. how to model the human observation, 2. how to estimate state using soft data and fuse it with the state estimates from the sensors on autonomous agents (e.g., a camera sensor). A novel approach is developed to model perceived human observations to the real physical states using artificial neural networks (ANN). A particle filter (PF) is used to estimate a moving target´s state based on range and bearing observation data from a human observer and an EKF is used to estimate the target state using on-board camera sensor. The range measurement is represented using Kumaraswamy´s double bounded distribution. The state estimates computed based on a model of human observation learned by an ANN are fused with the state estimates from the on-board sensors using a fast covariance intersection (CI) algorithm. The CI algorithm yields consistent fused estimates in the absence of unknown correlations between state estimates obtained using human measurements and robot sensor measurements. The performance of the developed algorithms is validated on a target tracking simulation platform.