A Decoupled Architecture for Scalability in Text Mining Applications

Sophisticated Text Mining features such as visualization, summarization, and clustering are becoming increasingly common in software applications. In Text Mining, documents are processed using techniques from different areas which can be very expensive in computation cost. This poses a scalability challenge for real-life applications in which users behavior can not be entirely predicted. This paper proposes a decoupled architecture for document processing in Text Mining applications, that allows applications to be scalable for large corpora and real-time processing. It contributes a software architecture designed around these requirements and presents TML, a Text Mining Library that implements the architecture. An experimental evaluation on its scalability using a standard corpus is also presented, and empirical evidence on its performance as part of an automated feedback system for writing tasks used by real students.