NLP based intelligent news search engine using information extraction from e-newspapers

Extracting text information from a web news page is a challenging task as most of the E-News content is provided with support from backend Content Management Systems (CMSs). In this paper, we present a personalized news search engine that focuses on building a repository of news articles by applying efficient extraction of text information from a web news page from varied e-news portals. The system is based on the concept of Document Object Model(DOM) tree manipulation for extracting text and modifying the web page structure to exclude irrelevant content like ads and user comments. We also use WordNet, a thesaurus of English language based on psycholinguist studies for matching the extracted content semantically to the title of the web page. TF-IDF (Term Frequency Inverse Document Frequency) is used for identifying the web page blocks carrying information relevant to the pages title. In addition to the extraction of information, functionalities to gather related information from different web news papers and to summarize the gathered information based on user preferences have also been included. We observed that the system was able to achieve good recall and high precision for both generalized and specific queries.