Effective Interpretation of Bucket Testing Results through Big Data Analytics

Bucket testing is a common practice in the internet industry, where new features and services are tested by exposing them to a randomly selected small subset of users. However, in this simple version of bucket testing, since a very small fraction of the total users are selected through uniform independent sampling of the population, the samples chosen, at times, do not adequately serve as a reasonable statistical proxy for the total population. This may lead to erroneous interpretation of the bucket testing results, particularly for online sites having large audiences with varying demographics and preferences. In this work, we present a novel algorithmic framework that addresses this challenge and provides an efficient and more accurate interpretation of the bucket testing results by analyzing the big audience data and factoring in the nature of the overall population in terms of the different user attributes. We demonstrate the effectiveness of our algorithm through the data obtained from real experiments conducted on Yahoo´s bucket testing platform.