Visual road assessment, which is carried out by many countries, involves the evaluation of millions of surface images. This exhaustive task is usually done manually and therefore is costly in terms of time and prone to failure. Different methods for automatic distress detection have been presented in the literature recently. However, most of the approaches are focused on crack detection only. This paper focuses on detecting multiple distress types and object classes on asphalt roads, aiming to fully automate distress detection on road surfaces in Austria, Switzerland, and Germany using image segmentation with neural networks. The paper introduces a distress and object catalog developed by experts of the involved countries that guarantees convertibility into federal distress catalogs. We evaluate the performance gain of different neural network architectures and advanced training techniques by conducting extensive experiments.