Data transmission with mean-square partial response

The authors investigate the use of mean-square-partial response (MSPR) channel signaling for symbol-by-symbol detection of a channel with controlled intersymbol interference. They show that, by filtering the received channel output with what they call a mean-square whitened matched filter, the overall detection SNR (signal-to-noise ratio) for symbol-by-symbol detection is as high as possible. In some cases, an uncoded improvement of 0.5-4.0 dB is possible with respect to existing partial response methods, referred to as zero-forcing partial response. A Tomlinson predecoder in concatenation with good trellis codes is used to combine the gain of the MSPR and trellis coding. It is shown that, with the MSPR system, optimization of the transmit filter leads to a system that is as close to its capacity as transmission on the flat AWGN (additive white Gaussian noise) channel is to its capacity. The authors also show that, when such a coded system has optimized transmit bandwidth, the cutoff rate is achievable with existing (most powerful) codes at an error rate 10^-6. A modified duobinary channel is used to illustrate the potential gains