Multi-dimensional and non-uniform constellation optimization via the special orthogonal group

With the goal of optimizing the CM (coded modulation) capacity of a finite constellation over a Rayleigh fading channel, we use one-parameter subgroups of the Lie group of rotation matrices to construct families of rotation matrices which optimize a certain objective function controlling the CM capacity. Our construction does not depend on any assumptions about the constellation or signal-to-noise ratio. We confirm the benefits of our construction for uniform and non-uniform constellations at a large range of SNR values through numerous simulations. We show that in two and four dimensions one can obtain a further potential increase in CM capacity by jointly considering non-uniform and rotated constellations.