Intersymbol interference with flat fading: Channel capacity

This paper finds the capacity of a linear time-invariant system with a given transfer function, observed in additive Gaussian noise through a memoryless fading channel. A coherent model is assumed where the fading coefficients are known at the receiver (but not the transmitter). We show that the optimum normalized power spectral density is the waterfilling solution for reduced signal-to-noise ratio, where the gap to the actual signal-to-noise ratio depends on both the fading distribution and the channel transfer function.