Development of a framework for sub-topic discovery from the Web

The motivation behind sub-topic or topic specific keyword discovery through Web pages is helping a user, who is insufficient in knowledge and experience about a topic, to find important concepts without much effort. Intuitively, a Web user would start searching the Web via querying search engines, visiting some pages, and spending a lot of time on deciding what is important about the topic and what is not. In this study, we try to mine important sub-topics or key concepts of a given topic automatically, through HTML based Web pages. Starting with a search query, the system gathers top-ranking pages returned from a search engine; and selects informative pages among them. These pages are processed further for extracting important phrases and then applied data mining techniques on these phrases to find candidate sub-topics. Each candidate phrase is given scores based on its relevance with the search query over the Web space. Using the proposed technique, the user should be able to quickly learn sub-topics or key concepts about a topic without going through the ordeal of browsing a large number of non-informative pages returned by the search engine.