Numerically efficient optimal design of cosine-modulated filter banks with peak-constrained least-squares behavior

Several current applications related to signal compression and representation and high-speed transmission require very selective filter banks/transmultiplexers. A possible solution is to employ the cosine-modulated filter banks/transmultiplexers (CMFBTs) where the prototype filters satisfy demanding constraints with respect to both the total stopband energy and maximum stopband ripple. This work proposes an efficient procedure to design nearly-perfect reconstruction CMFBT prototype filters with peak-constrained least-squares characteristics using a modified weighted least-squares algorithm. Substantial flexibility is added in the design of the magnitude response of the prototype filter, ranging from minimum stopband energy to minimum stopband ripple, which may be required in many applications. Some constraints are imposed to the CMFBTs in order to control the direct transfer and aliasing distortion functions, related to the intercarrier and intersymbol interferences. Algebraic simplifications are also provided on the overall objective function and associated constraints, leading to substantial reduction on the computational burden of the optimization process. The procedure is proven to be very powerful in designing CMFBT systems satisfying multiple constraints as indicated by numerical examples.