Exploring ‘hidden’ parts of the Web: The Hidden Web

A wealth of information is hidden within databases behind search interfaces that is unreachable to the traditional Web crawlers. One of the core challenges faced by a search engine is crawling this portion of the Web that is typically known as the Hidden Web. As the WWW continues to grow at an exponential rate, the problem of accurately retrieving information from the Hidden Web continues to exacerbate. This paper focuses on the approaches this content can be made visible to search engines and the various issues the crawlers come up with. The paper also articulates the tasks that need to be carried out by the crawler to overcome and patch up the issues.