Capacity penalty due to ideal zero-forcing decision-feedback equalization

We consider the capacity C of a continuous-time channel with frequency response H(f) and additive white Gaussian noise. If H(f)|^-2 behaves like a polynomial of order ρ at high frequencies, we show that the per-symbol capacity approaches ρ/2 nats per channel use at high signal powers. If the receiver uses an ideal zero forcing decision-feedback equalizer (DFE) consisting of a sampled whitened-matched filter followed by a zero-forcing tail canceler that is free of error propagation, the overall system is free of intersymbol interference and has a well-defined capacity C_ZF. By comparing this capacity with the capacity C of the underlying channel, we quantify the loss of information inherent in the tail-canceling operation that typifies zero-forcing DFE and zero-forcing precoding systems. For strictly bandlimited channels, we find that the capacity penalty approaches zero in the limit of large signal power. On the other hand, for nonstrictly bandlimited channels, the asymptotic penalty is nonzero; however, with bandwidth optimization, the asymptotic penalty is at most 0.59 dB, and the asymptotic ratio C_ZF/C is at least 93.6%, depending on the asymptotic order ρ of the channel response