Probability of a Segregating Pattern in a Sample of DNA Sequences

Mutations that result in segregating sites (polymorphic sites) in a sample of DNA sequences can be classified into different types. A pattern of segregating sites is an array of the numbers of various types of mutations. Using an urn model, the probability of a pattern of segregating sites can be expressed as a recurrence equation and its value can be computed sequentially. Among those that can be computed by this method are the probability of obtainingkexternal mutations (mutations that occur in external branches of the genealogy of a sample), the probability of obtainingkinternal mutations (mutations that occur in internal branches), the probability of obtainingksingletons (segregating sites at which one of the two segregating nucleotides is present in only one sequence), and the probability of obtainingknon-singletons. Two applications of the method are discussed. One is a maximum likelihood estimation ofθand another is a Bayesian statistical test of the hypothesis of neutral mutations.