A Reference Framework for the Automated Exploration of Web Applications

Web crawling is the process of exhaustively exploring the contents of a web site or application through automated means. While the results of such a crawling can be put through numerous uses ranging from a simple backup to comprehensive testing and analysis, features of modern-day applications prevent crawlers from properly exploring applications. We provide an in-depth analysis of 15 such features, and report on their presence in a study of 16 real-world web sites. Based on that study, we develop a configurable web application where the presence of each such feature can be turned on or off, aimed as a test bench where existing crawlers can be compared in a uniform way. Our results, which are the first exhaustive comparison of available crawlers, indicates areas where future work should be aimed.