Abstract

The Milky Way and a significant fraction of galaxies are observed to host a central massive black hole (MBH) embedded in a non-spherical nuclear star cluster. We study the secular orbital evolution of compact-object binaries in these environments and characterize the excitation of extremely large eccentricities that can lead to mergers by gravitational radiation. We find that the eccentricity excitation occurs most efficiently when the nodal precession timescale of the binary's orbit around the MBH due to the non-spherical cluster becomes comparable (within a factor of similar to 10) to the timescale on which the binary is torqued by the MBH due to the Lidov-Kozai (LK) mechanism. We show that in this regime the perturbations due to the cluster increase the fraction of systems that reach extreme eccentricities (1-e similar to 10(-4)-10(-6)) by a factor of similar to 10-100 compared to the idealized case of a spherical cluster, increasing the merger rates of compact objects by a similar factor. We identify two main channels that lead to this extreme eccentricity excitation: (i) chaotic diffusion of the eccentricities due to resonance overlap; (ii) cluster-driven variations of the mutual inclinations between the binary orbit and its center-of-mass orbit around the MBH, which can intensify the LK oscillations. We estimate that our mechanism can produce BH-BH and BH-neutron star binary merger rates of up to approximate to 15 Gpc(-3)yr(-1) and approximate to 0.4 Gpc(-3)yr(-1), respectively. Thus, we propose the cluster-enhanced LK mechanism as a new channel for the merger of compact-object binaries, competing with scenarios that invoke isolated binary evolution or dynamical formation in globular clusters.