Forbidden subsequences and Chebyshev polynomials Original Research Article

In (West, Discrete Math. 157 (1996) 363–374) it was shown using transfer matrices that the number |Sn(123; 3214)| of permutations avoiding the patterns 123 and 3214 is the Fibonacci number F2n (as are also |Sn(213; 1234)| and |Sn(213; 4123)|). We now find the transfer matrix for |Sn(123; r, r − 1,…,2, 1, r − 1)|, |Sn(213; 1,2,…,r,r + 1)|, and |Sn(213;r + 1, 1, 2,…,r)|, determine its characteristic polynomial in terms of the Chebyshev polynomials, and go on to determine the generating function as a quotient of modified Chebyshev polynomials. This leads to an asymptotic result for each r which collapses to the exact results 2n when r = 2 and F2n when r = 3 and to the Catalan number cn as r → ∞. We observe that our generating function also enumerates certain lattice paths, plane trees, and directed animals, giving hope that these areas of combinatorics can be applied to enumerating permutations with excluded subsequences.