The Design and Implement of High Efficient Incremental Microblogging Crawler

With the rapid development of microblog technology, many interesting research issues on microblog have aroused growing attention. Data fetching from microblog is the groundwork of these researches. In this paper we take Sina microblog (also called Weibo) as the crawling site, designing and implementing a high efficient incremental microblog crawler based on the classic multi-producers and multi-consumers model. Experimental results demonstrate that the crawler can collect real time microblog information efficiently and precisely.