Vision-Based Web Page Block Segmentation and Informative Block Detection

A web page usually contains various content such as main content block, navigation bars or columns, contacts information at the bottom, advertisements, or just some decoration components. Apart from the main content block, the other parts are all not related to the topic of the web page, we call them noise blocks. Content in noise blocks will seriously harm information extraction, web mining, we searching, etc. Identifying the main content block is a key issue. In this paper, we propose a new low-cost vision based web page segmentation and informative block detection algorithm. Then we apply this algorithm to develop a bids update detection system for a local bids collecting company. Our algorithm mainly utilizes the position information of HTML element, which lowers the cost of applying multiple rules in VIPS. Our proposed row-column splitting indicator helps us provide an easy-to-use partition granularity value which solves the difficulty of choosing an appropriate Degree of Coherence (DoC) value in VIPS algorithm. This row-column splitting indicator also avoids the traditional high cost clustering process in detecting informative block. Through extensive experiments, we conclude that our proposed algorithm has a comparable performance compared to the VIPS and other informative block detection algorithms, but with ease of use granularity value and lower computation cost.