Discovery, analysis and monitoring of hidden social networks and their evolution

Social networks that arise spontaneously and evolve over time have become an important component of ever growing global societies used for spreading ideas and indoctrinating people. Their loose membership and dynamics make them difficult to observe and monitor. We present a set of tools for discovery, analysis and monitoring evolution of hidden social groups on the internet and in cyberspace. Two complementary kinds of tools form a core of our approach. One is based on statistical analysis of communication network without considering communication content. The other focuses on communication content and analyzes recursive patterns arising in it. First, we present a software system SIGHTS (Statistical Identification of Groups Hidden in Time and Space), designed for the discovery, analysis, and knowledge visualization of social coalition in communication networks by analyzing communication patterns. We discuss how our algorithms extract groups and track their evolution in Enron-email dataset and in Blog data. The goal of SIGHTS is to assist an analyst in identifying relevant information. A complementary set of tools uses Recursive Data Mining (RDM) to identify frequent patterns in communication content such as email, blog or chat-room sessions. Our approach enables discovery of patterns at varying degrees of abstraction, in a hierarchical fashion, and in language independent way. We use RDM to distinguish among different roles played by communicators in social networks (e.g., distinguishing between leaders and members). Experiments on the Enron dataset, which categorize members into organizational roles demonstrate that use of the RDM dominant patterns improves role detection.