This paper proposes a new framework for ground extraction and building detection in LiDAR data. The proposed approach constructs the connectivity of a grid over the LiDAR point-cloud in order to perform multi-scale data decomposition. This is realised by forming a top-hat scale-space using differential morphological profiles (DMPs) on points’ residuals from the approximated surface. The geometric attributes of the contained features are estimated by mapping characteristic values from DMPs. Ground definition is achieved by using features’ geometry, whilst their surface and regional attributes are additionally considered for building detection. A new algorithm for local fitting surfaces (LoFS) is proposed for extracting planar points. Finally, transitions between planar ground and non-ground regions are observed in order to separate regions of similar geometrical and surface properties but different contexts (i.e. bridges and buildings). The methods were evaluated using ISPRS benchmark datasets and show superior results in comparison to the current state-of-the-art.