Learning based web crawl forum

The main objective of this project is to crawl applicable forum content from the web with minimal overhead. Forum threads usually contain the information content that is the target of the forum crawlers. The system that is to be created for learn URL patterns across multiple sites and automatically finds a forum´s entry page given a page from the forum. The forum has different layouts, styles and a generic crawler that blindly follows the duplicate links and uninformative page will crawl duplicate pages. The test results will show that the proposed system achieved effectiveness and coverage on a large set of test forums.