OGSA-DWC: A Middleware for Deep Web Crawling Using the Grid

Conventional search engines generally cannot find information from the Deep Web because they use hyper link-based crawling techniques to visit Web pages. Recently, lots of research efforts are being tried to crawl the Deep Web. One of the obstacles for crawling the Deep Web is the requirement of huge computing resources, but most of search engine companies hardly meet the needs. We, therefore, propose the design of the Grid-based middleware, OGSA-DWC for crawling the Deep Web. With our middleware, developers will easily implement a Grid-based Deep Web crawling system although they do not have much knowledge about how to use idle and distributed computing resources.