We propose a method for cycle-slip mitigation in circular multilevel quadrature amplitude modulation (C-mQAM) coherent optical systems, with constellation rotational asymmetry, based on an adaptive boundaries approach. The impact of cycle-slips in C-mQAM coming from Viterbi–Viterbi algorithm limits the phase noise tolerance. By introducing adaptive boundaries and a differential coding scheme, the ambiguity of asymmetrical rotation of constellation can be effectively removed. Performance of the proposed method is evaluated for a C-16QAM and C-64QAM for various laser linewidths. Results show a noticeable improvement in linewidth symbol duration product (
tolerance compared with the previous studies on C-mQAM and mQAM constellations. The
tolerance reaches
and
for C-16QAM and C-64QAM, respectively, for 1 dB penalty at a symbol error rate of
.