We study the discovery prospect of different three body lepton number violating (LNV) meson decays $M_1^{-}\to {\ell }_1^{-}{\ell }_2^{-}{M}_2^{+}$ in the framework of the right-handed (RH) neutrino extended Standard Model. We consider a number of ongoing experiments, such as NA62 and LHCb at CERN, Belle II at SuperKEK, as well as at the proposed future experiments, SHiP, MATHUSLA and FCC-ee. The RH Majorana neutrino $N$ mediating these meson decays provides a resonant enhancement of the rates, if the mass of $N$ lies in the range (100 MeV–6 GeV). We consider the effect of parent mesons velocity, as well as the effect of finite detector size. Using the expected upper limits on the number of events for the LNV decay modes, $M_1^{-}\to {\ell }_1^{-}{\ell }_2^{-}{\pi }^{+}$ ($M_1=B,B_c,D,D_s$ and $K$), we analyze the sensitivity reach of the mixing angles $|V_{eN}{|}^2$, $|V_{\mu N}{|}^2$, $|V_{\tau N}{|}^2$, $|V_{eN}{V}_{\mu N}|$, $|V_{eN}{V}_{\tau N}|$ and $|V_{\mu N}{V}_{\tau N}|$ as a function of heavy neutrino mass $M_N$. We show that inclusion of parent meson velocity can account to a large difference for active-sterile mixing, especially for $D$, $D_s$ meson decay at SHiP and $K$ meson decay at NA62. Taking into account the velocity of the $D_s$ meson, the future beam dump experiment SHiP can probe $|V_{eN}{|}^2\sim {10}^{-9}$. For RH neutrino mass in between 2–5 GeV, MATHUSLA can provide the best sensitivity reach of active-sterile mixings.

• Received 5 September 2019

DOI:https://doi.org/10.1103/PhysRevD.100.095022

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI. Funded by SCOAP³.

Published by the American Physical Society