Kin selection and coefficients of relatedness in family-structured populations with inbreeding

We consider family specific fitnesses that depend on mixed strategies of two basic phenotypes or behaviours. Pairwise interactions are assumed, but they are restricted to occur between sibs. To study the change in frequency of a rare mutant allele, we consider two different forms of weak selection, one applied through small differences in genotypic values determining individual mixed strategies, the other through small differences in viabilities according to the behaviours chosen by interacting sibs. Under these two specific forms of weak selection, we deduce conditions for initial increase in frequency of a rare mutant allele for autosomal genes in the partial selfing model as well as autosomal and sex-linked genes in the partial sib-mating model with selection before mating or selection after mating. With small differences in mixed strategies, we show that conditions for protection of a mutant allele are tantamount to conditions for initial increase in frequency obtained in additive kin selection models. With particular reference to altruism versus selfishness, we provide explicit ranges of values for the selfing or sib-mating rate based on a fixed cost–benefit ratio and the dominance scheme that allow the spreading of a rare mutant allele into the population. This study confirms that more inbreeding does not necessarily promote the evolution of altruism. Under the hypothesis of small differences in viabilities, the situation is much more intricate unless an additive model is assumed. In general however, conditions for initial increase in frequency of a mutant allele can be obtained in terms of fitness effects that depend on the genotypes of interacting individuals or their mates and generalized conditional coefficients of relatedness according to the inbreeding condition of the interacting individuals.