Error Rate Performance of Multilevel Signals with Coherent Detection

In this paper, coherent detection for multilevel correlated signaling sets in additive white Gaussian noise is addressed. The contribution is twofold. Firstly, correlation structures that minimize the symbol error probability (SEP) of M-ary frequency-shift keying as well as of arbitrarily correlated signaling sets are investigated, while secondly, a general and analytical expression for the SEP is derived in the form of a single integral. The structure of the associated correlation matrix is generic and includes various known signaling sets as special cases. Specific correlation structures that minimize the SEP are also studied. Based on eigendecomposition or LU decomposition, generic methods for constructing a correlated signaling set for any correlation matrix under consideration are also provided.