Intersymbol interference and error probability

Using a criterion of minimum average error probability we derive a method for specifying an optimum linear, time invariant receiving filter for a digital data transmission system. The transmitted data are binary and coded into pulses of shape . The linear transmission medium introduces intersymbol interference and additive Gaussian noise. Because the intersymbol interference is not Gaussian and can be correlated with the binary digit being detected, our problem is one of deciding which of two waveforms is present in a special type of correlated, non-Gaussian noise. For signal-to-noise ratios in a range of practical interest, the optimum filter is found to be representable as a matched filter followed by a tapped delay line--the same form as that of the least mean square estimator of the pulse amplitude. The performance (error probability vs. ) of the optimum filter is compared with that of a matched-filter receiver in an example.