The edge of the M87 halo and the kinematics of the diffuse light in the Virgo cluster core

Aims. We study the kinematics and dynamics of the extreme outer halo of M87, the central galaxy in the Virgo cluster, and its transition to the intracluster light (ICL). Methods. We present high resolution FLAMES/VLT spectroscopy of intracluster planetary nebula (PN) candidates, targeting three new fields in the Virgo cluster core with surface brightness down to mu(B) = 28.5. Based on the projected phase space information ( sky positions and line-of-sight velocities) we separate galaxy and cluster components in the confirmed PN sample. We then use the spherical Jeans equation and the total gravitational potential as traced by the X-ray emission to derive the orbital distribution in the outer stellar halo of M87. We determine the luminosity-specific PN number for the M87 halo and the ICL from the photometric PN catalogs and sampled luminosities, and discuss the origin of the ICL in Virgo based on its measured PN velocities. Results. We confirm a further 12 PNs in Virgo, five of which are bound to the halo of M87, and the remainder are true intracluster planetary nebulas (ICPNs). The M87 PNs are confined to the extended stellar envelope of M87, within a projected radius of similar to 160 kpc, while the ICPNs are scattered across the whole surveyed region between M87 and M86, supporting a truncation of M87's luminous outer halo at a 2s level. The line-of-sight velocity distribution of the M87 PNs at projected radii of 60 kpc and 144 kpc shows (i) no evidence for rotation of the halo along the photometric major axis; and (ii) that the velocity dispersion decreases in the outer halo, down to sigma(last) = 78 +/- 25 km s(-1) at 144 kpc. The Jeans model for the M87 halo stars fits the observed line-of-sight velocity dispersion profile only if the stellar orbits are strongly radially anisotropic (beta similar or equal to 0.4 at r similar or equal to 10 kpc increasing to 0.8 at the outer edge), and if additionally the stellar halo is truncated at similar or equal to 150 kpc average elliptical radius. The alpha-parameters for the M87 halo and the ICL are in the range of values observed for old (> 10 Gyr) stellar populations. Conclusions. Both the spatial segregation of the PNs at the systemic velocity of M87 and the dynamical model support that the stellar halo of M87 ends at similar to 150 kpc. We discuss several possible explanations for the origin of this truncation but are unable to discriminate between them: tidal truncation following an earlier encounter of M87 with another mass concentration in the Virgo core, possibly around M84, early AGN feedback effects, and adiabatic contraction due to the cluster dark matter collapsing onto M87. From the spatial and velocity distribution of the ICPNs we infer that M87 and M86 are falling towards each other and that we may be observing them just before the first close pass. The new PN data support the view that the core of the Virgo cluster is not yet virialized but is in an ongoing state of assembly, and that massive elliptical galaxies are important contributors to the ICL in the Virgo cluster.