A design of optimal overlay system

In this paper, the optimal design of a wideband overlay system is considered for communications over a white Gaussian noise channel corrupted by the signal from an existing system. Under the assumptions that the symbol transmission rate of the new system is an integer multiple of that of the existing system and that the transmit filter is time invariant, the optimum transmitter and receiver waveforms are derived that jointly minimize the average mean-squared error (MSE) at the output of the linear receiver of the new system. It is shown that this joint optimization technique leads to significant performance improvement through complete interference avoidance over the overlay systems with receiver-only optimization, and over those with no transmitter and receiver optimization. It is also shown that, if the existing system does not induce any self interference, the derived solution reduces to the generalized Nyquist condition for no mutual as well as self interference. The case where the symbol transmission rate of the existing system is an integer multiple of that of the new system is also briefly discussed.