On the instability of evolutionary stable strategies in small populations

Evolutionary stable strategies (ESSs) are often used to explain the behaviors of individuals and species. The analysis of ESSs determines which, if any, combinations of behaviors cannot be invaded by alternative strategies. Two assumptions required to generate an ESS (i.e. an infinite population and payoffs described only on the average) do not hold under natural conditions. Previous experiments have indicated that under more realistic conditions of finite populations and stochastic payoffs, populations may evolve in trajectories that are unrelated to an ESS, even in very simple evolutionary games. The simulations are extended here to small populations with varying levels of selection pressure and mixing levels. The results suggest that ESSs may not provide a good explanation of the behavior of small populations even at relatively low levels of selection pressure and even under persistent mixing. The implications of these results are discussed briefly in light of previous literature which claimed that ESSs generated suitable explanations of real-world data.