Competition between Heisenberg and bond-dependent interactions driven an incommensurate ordering in honeycomb cobaltates

Experimental observations show that the zero-field ground state of honeycomb-layered cobaltates is an incommensurate spiral order with an ordering wave vector along the line. However, in the presence of a magnetic field, the incommensurate magnetic phase becomes unstable and the quantum spin liquid phase becomes stable in this class of materials. Here, the central question we address is how to realize the incommensurate order theoretically. For this purpose, we are interested in studying the generalized Kitaev-Heisbenerg model and investigating the interplay effect between conventional Heisenberg and bond-dependent exchange interactions on the stability of the incommensurate spiral order. Our results show that the interplay between the second neighbor Heisenberg interaction and bond-dependent exchange interactions play a vital role in the stability of incommensurate phases. The competition between the second nearest nearest-neighbor interaction and the off-diagonal bond-directional exchange for  may place this material in proximity to a quantum spin liquid phase. The quantum spin liquid can be accessible for  in the presence of an external magnetic field.