Showing 1–8 of 8 results for author: Petej, I

Protected probabilistic classification

Authors: Vladimir Vovk, Ivan Petej, Alex Gammerman

Abstract: This paper proposes a way of protecting probabilistic prediction models against changes in the data distribution, concentrating on the case of classification and paying particular attention to binary classification. This is important in applications of machine learning, where the quality of a trained prediction algorithm may drop significantly in the process of its exploitation. Our techniques are… ▽ More This paper proposes a way of protecting probabilistic prediction models against changes in the data distribution, concentrating on the case of classification and paying particular attention to binary classification. This is important in applications of machine learning, where the quality of a trained prediction algorithm may drop significantly in the process of its exploitation. Our techniques are based on recent work on conformal test martingales and older work on prediction with expert advice, namely tracking the best expert. △ Less

Submitted 22 October, 2021; v1 submitted 4 July, 2021; originally announced July 2021.

Comments: 23 pages, 14 figures, and 4 tables

MSC Class: 68Q32 (Primary) 68T05; 60G25; 60G42; 62F03; 62M20 (Secondary)

Retrain or not retrain: Conformal test martingales for change-point detection

Authors: Vladimir Vovk, Ivan Petej, Ilia Nouretdinov, Ernst Ahlberg, Lars Carlsson, Alex Gammerman

Abstract: We argue for supplementing the process of training a prediction algorithm by setting up a scheme for detecting the moment when the distribution of the data changes and the algorithm needs to be retrained. Our proposed schemes are based on exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the data. Our method, based on conformal prediction, is… ▽ More We argue for supplementing the process of training a prediction algorithm by setting up a scheme for detecting the moment when the distribution of the data changes and the algorithm needs to be retrained. Our proposed schemes are based on exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the data. Our method, based on conformal prediction, is general and can be applied on top of any modern prediction algorithm. Its validity is guaranteed, and in this paper we make first steps in exploring its efficiency. △ Less

Submitted 20 February, 2021; originally announced February 2021.

Comments: 22 pages, 19 figures, 3 tables

MSC Class: 68Q32 (Primary) 62G10; 60G42; 68T05 (Secondary)

Computationally efficient versions of conformal predictive distributions

Authors: Vladimir Vovk, Ivan Petej, Ilia Nouretdinov, Valery Manokhin, Alex Gammerman

Abstract: Conformal predictive systems are a recent modification of conformal predictors that output, in regression problems, probability distributions for labels of test observations rather than set predictions. The extra information provided by conformal predictive systems may be useful, e.g., in decision making problems. Conformal predictive systems inherit the relative computational inefficiency of conf… ▽ More Conformal predictive systems are a recent modification of conformal predictors that output, in regression problems, probability distributions for labels of test observations rather than set predictions. The extra information provided by conformal predictive systems may be useful, e.g., in decision making problems. Conformal predictive systems inherit the relative computational inefficiency of conformal predictors. In this paper we discuss two computationally efficient versions of conformal predictive systems, which we call split conformal predictive systems and cross-conformal predictive systems. The main advantage of split conformal predictive systems is their guaranteed validity, whereas for cross-conformal predictive systems validity only holds empirically and in the absence of excessive randomization. The main advantage of cross-conformal predictive systems is their greater predictive efficiency. △ Less

Submitted 3 November, 2019; originally announced November 2019.

Comments: 31 pages, 14 figures, 1 table. The conference version published in the Proceedings of COPA 2018, and the journal version is to appear in Neurocomputing

MSC Class: 68T05

Conformal calibrators

Authors: Vladimir Vovk, Ivan Petej, Paolo Toccaceli, Alex Gammerman

Abstract: Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems that are fully adaptive. Our method consists in calibrating existing predictive sys… ▽ More Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems that are fully adaptive. Our method consists in calibrating existing predictive systems; the input predictive system is not supposed to satisfy any properties of validity, whereas the output predictive system is guaranteed to be calibrated in probability. It is interesting that the method may also work without the IID assumption, standard in conformal prediction. △ Less

Submitted 18 February, 2019; originally announced February 2019.

Comments: 10 pages, 2 figures

Report number: 23 MSC Class: 68T05 ACM Class: I.2.6

Criteria of efficiency for conformal prediction

Authors: Vladimir Vovk, Ilia Nouretdinov, Valentina Fedorova, Ivan Petej, Alex Gammerman

Abstract: We study optimal conformity measures for various criteria of efficiency of classification in an idealised setting. This leads to an important class of criteria of efficiency that we call probabilistic; it turns out that the most standard criteria of efficiency used in literature on conformal prediction are not probabilistic unless the problem of classification is binary. We consider both unconditi… ▽ More We study optimal conformity measures for various criteria of efficiency of classification in an idealised setting. This leads to an important class of criteria of efficiency that we call probabilistic; it turns out that the most standard criteria of efficiency used in literature on conformal prediction are not probabilistic unless the problem of classification is binary. We consider both unconditional and label-conditional conformal prediction. △ Less

Submitted 14 September, 2016; v1 submitted 14 March, 2016; originally announced March 2016.

Comments: 31 pages

MSC Class: 68T05 ACM Class: I.2.6

Large-scale probabilistic predictors with and without guarantees of validity

Authors: Vladimir Vovk, Ivan Petej, Valentina Fedorova

Abstract: This paper studies theoretically and empirically a method of turning machine-learning algorithms into probabilistic predictors that automatically enjoys a property of validity (perfect calibration) and is computationally efficient. The price to pay for perfect calibration is that these probabilistic predictors produce imprecise (in practice, almost precise for large data sets) probabilities. When… ▽ More This paper studies theoretically and empirically a method of turning machine-learning algorithms into probabilistic predictors that automatically enjoys a property of validity (perfect calibration) and is computationally efficient. The price to pay for perfect calibration is that these probabilistic predictors produce imprecise (in practice, almost precise for large data sets) probabilities. When these imprecise probabilities are merged into precise probabilities, the resulting predictors, while losing the theoretical property of perfect calibration, are consistently more accurate than the existing methods in empirical studies. △ Less

Submitted 13 November, 2015; v1 submitted 1 November, 2015; originally announced November 2015.

Comments: 38 pages, 14 figures, to appear in Advances in Neural Information Processing Systems 28 (NIPS 2015). As compared with the previous version (v1), the MATLAB code (the 5 files with extension .m) and results of new empirical studies have been added

Report number: 13 MSC Class: 68T05

From conformal to probabilistic prediction

Authors: Vladimir Vovk, Ivan Petej, Valentina Fedorova

Abstract: This paper proposes a new method of probabilistic prediction, which is based on conformal prediction. The method is applied to the standard USPS data set and gives encouraging results. This paper proposes a new method of probabilistic prediction, which is based on conformal prediction. The method is applied to the standard USPS data set and gives encouraging results. △ Less

Submitted 21 June, 2014; originally announced June 2014.

Comments: 12 pages, 2 tables

MSC Class: 68T10

Venn-Abers predictors

Authors: Vladimir Vovk, Ivan Petej

Abstract: This paper continues study, both theoretical and empirical, of the method of Venn prediction, concentrating on binary prediction problems. Venn predictors produce probability-type predictions for the labels of test objects which are guaranteed to be well calibrated under the standard assumption that the observations are generated independently from the same distribution. We give a simple formaliza… ▽ More This paper continues study, both theoretical and empirical, of the method of Venn prediction, concentrating on binary prediction problems. Venn predictors produce probability-type predictions for the labels of test objects which are guaranteed to be well calibrated under the standard assumption that the observations are generated independently from the same distribution. We give a simple formalization and proof of this property. We also introduce Venn-Abers predictors, a new class of Venn predictors based on the idea of isotonic regression, and report promising empirical results both for Venn-Abers predictors and for their more computationally efficient simplified version. △ Less

Submitted 21 June, 2014; v1 submitted 31 October, 2012; originally announced November 2012.

Comments: 18 pages; to appear in the UAI 2014 Proceedings

Report number: OCM07 MSC Class: 68T05; 68T10