The Effect of Neutral Nonadditive Gene Action on the Quantitative Index of Population Divergence

For neutral additive genes, the quantitative index of population divergence (QST) is equivalent to Wrightʹs fixation index (FST). Thus, divergent or convergent selection is usually invoked, respectively, as a cause of the observed increase (QST > FST) or decrease (QST < FST) of QST from its neutral expectation (QST = FST). However, neutral nonadditive gene action can mimic the additive expectations under selection. We have studied theoretically the effect of consecutive population bottlenecks on the difference FST - QST for two neutral biallelic epistatic loci, covering all types of marginal gene action. With simple dominance, QST < FST for only low to moderate frequencies of the recessive alleles; otherwise, QST > FST. Additional epistasis extends the condition QST < FST to a broader range of frequencies. Irrespective of the type of nonadditive action, QST < FST generally implies an increase of both the within-line additive variance after bottlenecks over its ancestral value (VA) and the between-line variance over its additive expectation (2FSTVA). Thus, both the redistribution of the genetic variance after bottlenecks and the FST - QST value are governed largely by the marginal properties of single loci. The results indicate that the use of the FST - QST criterion to investigate the relative importance of drift and selection in population differentiation should be restricted to pure additive traits.