Transversality of homoclinic orbits to hyperbolic equilibria in a Hamiltonian system, via the Hamilton–Jacobi equation

We consider a Hamiltonian system with 2 degrees of freedom, with a hyperbolic equilibrium point having a loop or homoclinic orbit (or, alternatively, two hyperbolic equilibrium points connected by a heteroclinic orbit), as a step towards understanding the behavior of nearly-integrable Hamiltonians near double resonances. We provide a constructive approach to study whether the unstable and stable invariant manifolds of the hyperbolic point intersect transversely along the loop, inside their common energy level. For the system considered, we establish a necessary and sufficient condition for the transversality, in terms of a Riccati equation whose solutions give the slope of the invariant manifolds in a direction transverse to the loop. The key point of our approach is to write the invariant manifolds in terms of generating functions, which are solutions of the Hamilton–Jacobi equation. In some examples, we show that it is enough to analyze the phase portrait of the Riccati equation without solving it explicitly. Finally, we consider an analogous problem in a perturbative situation. If the invariant manifolds of the unperturbed loop coincide, we have a problem of splitting of separatrices. In this case, the Riccati equation is replaced by a Melnikov potential defined as an integral, providing a condition for the existence of a perturbed loop and its transversality. This is also illustrated with a concrete example.