Asymptotic test of mixture model and its applications to QTL interval mapping

Quantitative trait loci (QTL) mapping has been a standard means in identifying genetic regions harboring potential genes underlying complex traits. Likelihood ratio test (LRT) has been commonly applied to assess the significance of a genetic locus in a mixture model content. Given the time constraint in commonly used permutation tests to assess the significance of LRT in QTL mapping, we study the behavior of the LRT statistic in mixture model when the proportions of the distributions are unknown. We found that the asymptotic null distribution is stationary Gaussian process after suitable transformation. The result can be applied to one-parameter exponential family mixture model. Under certain condition, such as in a backcross mapping model, the tail probability of the supremum of the process is calculated and the threshold values can be determined by solving the distribution function. Simulation studies were performed to evaluate the asymptotic results.