Improving web data extraction by noise removal

Internet is the largest information repository consisting of large volume of data on almost every arena. As the websites are getting more and more complex the demand to make the web page adaptive according to the web user and hardware platform has become more challenging. There exist various web extraction systems but the varied requirement has given a challenge to the researchers to focus on devising new methodology to perform the task of information extraction more accurately and specific to the need of the user. In our work we have focused on the area of noise removal from the web page, as after removing the noise from the web page, the performance of web mining technique like classification, clustering and search engine web crawling etc. improves by many folds. We have developed an algorithm to extract the Visual Blocks of a Web page of a web site using DOM and Visual Characteristics, and then it is converted to the Pattern Tree. The Pattern Tree of different web pages of a single web site is mapped to find the similarity pattern among the web pages. For each node the Node Importance Measure is calculated, which is used to discriminate noise and main element of the web page. It is generally observed that Web Pages of a single web site often follow similar layout pattern, and the noise elements are repeated in almost all web pages.