SERPWatcher: A Sophisticated Mining Tool Utilizing Search Engine Results Pages (SERPs) for Social Change Discovery

As various individuals and organizations disseminate information on their Web pages, real-world social events and changes are considered to be reflected in Web trends. The billions of Web pages that now exist are retrieved by Web search engines which accept keywords and return a search engine results page (SERP). Since the SERP itself and the ranking order change with time reflecting the changes in society, it might be possible to accurately follow the movement of society by mining SERPs. This paper reports the design and implementation of a SERP mining tool named SERPWatcher. It provides sophisticated interfaces and functions for SERP miners in the field of social sciences to discover social changes. It could be a novel social survey method in that it totally differs from the traditional methods such as questionnaires and interviews. A research prototype of SERPWatcher is currently under operation, and its validation test shows that it has a fail-safe nature in the sense that social changes are mirrored on the changes of the SERP ranking order. Also, the testers conducting gender studies have been expressing positive opinions on the use of SERPWatcher as a novel research methodology in the field of sociology.