Distributed Data Mining on Virtual Clusters

For complex processes investigated in scientific fields such as medicine and earth sciences, knowledge discovery that exposes the underlying structure of the processes is crucial for detecting changes of state and constructing forecasting procedures. A model discovery approach has been recently developed, which uses computational intelligence techniques to deal with the heterogeneity, incompleteness and imprecision of the data describing complex proceeses. While the approach offers a tractable and effective means for model discovery, it is still computationally expensive, routinely requiring tens of thousands of hours of CPU time. To satisfy the needs of such applications, it is more costeffective to employ shared resources located in different departments of an organization, than to purchase large and expensive compute clusters. We present a method for aggregating resources under multiple administrative domains into a virtual resource that can satisfy efficiently the needs of data-mining based model discovery. The proposed resource aggregation and job management approach provides an end-to-end solution to distributed data mining across organization-wide resources.