To reject or not to reject: that is the question-an answer in case of neural classifiers

A method defining a reject option that is applicable to a given 0-reject classifier is proposed. The reject option is based on an estimate of the classification reliability, measured by a reliability evaluator Ψ. Trivially, once a reject threshold σ has been fixed, a sample is rejected if the corresponding value of Ψ is below σ. Obviously, as σ represents the least tolerable classification reliability level, when its value varies the reject option becomes more or less severe. In order to adapt the behavior of the reject option to the requirements of the considered application domain, a function P characterizing the reject option's adequacy to the domain has been introduced. It is shown that P can be expressed as a function of σ and, consequently, the optimal value for σ is defined as the one which maximizes the function P. The method for determining the optimal threshold value is independent of the specific 0-reject classifier, while the definition of the reliability evaluators is related to the classifier's architecture. General criteria for defining appropriate reliability evaluators within a classification paradigm are illustrated in the paper and are based on the localization, in the feature space, of the samples that could be classified with a low reliability. The definition of the reliability evaluators for three popular architectures of neural networks (backpropagation, learning vector quantization and probabilistic network) is presented. Finally, the method has been tested with reference to a complex classification problem with data generated according to a distribution-of-distributions model