Joint optimization of transmitter and receiver filters in digital PAM systems with a Viterbi detector

Joint optimization of transmitter and receiver filters in a pulse-amplitude modulation (PALM) system with an arbitrary but fixed Viterbi detector is considered. Optimization is performed under an average transmitted power constraint with respect to a new criterion called the effective mean-square error (EMSE). It is shown that the optimization problem is convex. A necessary and sufficient condition for optimality is derived, and a method for optimization is proposed. Numerical results are obtained for a channel with coaxial cable characteristics.