A/B Testing at SweetIM: The Importance of Proper Statistical Analysis

A/B tests are an important aspect of modern applications, particularly on the internet, as they allow businesses to optimize their user or customer experience to maximize usage and ultimately profits. In this paper we review Sweet IM A/B testing system, with focus on statistical aspects and methodology. We describe all parameters of the Sweet IM A/B test environment including randomization mechanism, data collection and analysis. We also provide some interesting examples and case studies from real A/B tests that took place at Sweet IM. Analyses of count data like number of searches or content sent per user are the most significant performance indicators for the majority of Sweet IM tests. Accuracy of such analyses is a key-success factor and can increase the ROI of the tests dramatically. We expand the popular method of the analyses of counts based on Poisson distribution and show its inappropriateness when dealing with over dispersed Poisson. We propose to use Negative Binomial distribution as appropriate solution for over dispersion in search and content count data. We show that the conclusions from analyses of specific A/A and A/B tests run in this application with NB differ from those with an incorrect Poisson assumption.