Towards hard+soft data fusion: Processing architecture and implementation for the joint fusion and analysis of hard and soft intelligence data

Historically, data fusion has focused on processing hard or physical sensor data while soft or human observed data has been neglected within fusion processes. This human observed data has much to offer towards obtaining comprehensive situational awareness, particularly in a domain such as intelligence analysis where subtle connections and interactions are difficult to observe with physical sensors. This paper describes the processing architecture designed and implemented for the fusion of hard and soft data in the multi-university research initiative on network-based hard and soft information fusion. The processing elements designed to successfully fuse and reason over the hard and soft data include the natural language processing elements to form propositional graphs from linguistic observations, conversion of the propositional graphs to attributed graphical form, alignment and tagging of the uncertainties extant in the human observations, conversion of hard data tracks to a graphical format, association of entities and relations in observational hard and soft data graphs and the matching of situations of interest to the cumulative data or evidential graph. To illustrate these processing elements within the integrated processing architecture a small synthetic data set entitled the bomber buster scenario is utilized, presenting examples of each processing element along the processing flow. The value of fusing hard and soft information is illustrated by demonstrating that individually, neither hard nor soft information could provide the situation estimate.