Neutron scattering investigation of proposed Kosterlitz-Thouless transitions in the triangular-lattice Ising antiferromagnet TmMgGaO4

The transverse-field Ising model on the triangular lattice is expected to host an intermediate finite-Temperature Kosterlitz-Thouless (KT) phase through a mapping of the spins on each triangular unit to a complex order parameter. TmMgGaO4 is a candidate material to realize such physics due to the non-Kramers nature of the Tm3+ ion and the resulting two-singlet single-ion ground state. Using inelastic neutron scattering, we confirm this picture by determining the leading parameters of the low-energy effective Hamiltonian of TmMgGaO4. Subsequently, we track the predicted KT phase and related transitions by inspecting the field and temperature dependence of the ac susceptibility. We further probe the spin correlations in both reciprocal space and real space via single-crystal neutron diffraction and magnetic total scattering techniques, respectively. Magnetic pair distribution function analysis provides evidence for the formation of vortex-Antivortex pairs that characterize the proposed KT phase around 5 K. Although structural disorder influences the field-induced behavior of TmMgGaO4, the magnetism in zero field appears relatively free from these effects. These results position TmMgGaO4 as a strong candidate for a solid-state realization of KT physics in a dense spin system.