An improved ideagraph algorithm for discovering important rare events

In recent years, Chance Discovery as an extension of data mining has been presented to discover rare but significant events, i.e., chances, for human decision making from large amounts of data. KeyGraph or IdeaGraph as a chance mining algorithm can capture these chances by converting the unstructured data into a scenario graph. However, they both fail to eliminate the interference of frequent events when uncovering rare events, causing a bottleneck of capturing important rare events. In this paper, we propose an improved algorithm of IdeaGraph to address this issue. It takes rare events as the essential components to preserve them from being filtered when forming a cluster. On base of that, it conducts cluster refining such as pruning and ranking to optimize the construction of a scenario graph. Additionally, it provides an enhanced method to evaluate important rare events by measuring the significance of an event on the perspectives of the co-occurring frequency and the probability distribution. An experiment demonstrates the superiority of our algorithm on capturing important rare events by comparing with benchmark algorithms.