Fast simulation of the energy depositions in high-granular detectors is needed for future collider experiments at ever-increasing luminosities. Generative machine learning (ML) models have been shown to speed up and augment the traditional simulation chain in physics analysis. However, the majority of previous efforts were limited to models relying on fixed, regular detector readout geometries. A major advancement is the recently introduced CaloClouds model, a geometry-independent diffusion model, which generates calorimeter showers as point clouds for the electromagnetic calorimeter of the envisioned International Large Detector (ILD). In this work, we introduce CaloClouds II which features a number of key improvements. This includes continuous time score-based modelling, which allows for a 25-step sampling with comparable fidelity to CaloClouds while yielding a 6× speed-up over Geant4 on a single CPU (5× over CaloClouds). We further distill the diffusion model into a consistency model allowing for accurate sampling in a single step and resulting in a 46× speed-up over Geant4(37× over CaloClouds). This constitutes the first application of consistency distillation for the generation of calorimeter showers.