Collective magnetic excitation modes, magnons, can be coherently coupled to microwave photons in the single excitation limit. This allows for access to quantum properties of magnons and opens up a range of applications in quantum information processing, with the intrinsic magnon linewidth representing the coherence time of a quantum resonator. Our measurement system consists of a yttrium iron garnet sphere and a three-dimensional microwave cavity at temperatures and excitation powers typical for superconducting quantum circuit experiments. We perform spectroscopic measurements to determine the limiting factor of magnon coherence at these experimental conditions. Using the input-output formalism, we extract the magnon linewidth . We attribute the limitations of the coherence time at lowest temperatures and excitation powers to incoherent losses into a bath of near-resonance two-level systems (TLSs), a generic loss mechanism known from superconducting circuits under these experimental conditions. We find that the TLSs saturate when increasing the excitation power from quantum excitation to multiphoton excitation and their contribution to the linewidth vanishes. At higher temperatures, the TLSs saturate thermally and the magnon linewidth decreases as well.