Clustering techniques for rule extraction from unstructured text fragments

This paper focuses on techniques for clustering unstructured text fragments which are generated from a rule extraction agent. The text fragments represent paragraphs of text containing potential rules. The latent semantic indexing method is applied to map the unstructured text into a linear vector space. Similar text fragments are identified based on the similarity between their vector representations. The problem of clustering based on general similarity measures that are different than the conventional distance based measures is discussed. A new version of the mountain clustering method is developed to address the problem of identifying groups of similar vectors that correspond to documents with analogous content. Several clustering algorithms are compared in their ability to satisfactorily cluster these text fragments into sets of related concepts. An intelligent agent algorithm for extraction of rules from text documents is proposed and demonstrated.