Objective evaluation of Spider Detection Techniques

Spider is a program for harvesting internet resources. Spiders Detection Techniques(SDT) are used to regulate and monitor behaviors of spiders visiting website. In this paper, an Evaluation Method based on Trap technique(EMT) is proposed to calculate the recall rate and precision rate of SDT. Without relying on manual analysis, it is more objective and more adaptive to the development of SDT. The principles of EMT bases on the statistical hypothesis that the distribution of users captured by trap obeys binomial distribution theory. The experiment of EMT indicates three conclusions: (1)EMT has the consistent conclusion with the manual analysis result. (2)EMT is little impacted by time span of analysis.(3)EMT is little impacted by the traps layout rate when it changes in ±10%.