TabularPredictor.calibrate_decision_threshold

TabularPredictor.calibrate_decision_threshold(data: str | TabularDataset | DataFrame | None = None, metric: str | Scorer | None = None, model: str = 'best', decision_thresholds: int | List[float] = 25, secondary_decision_thresholds: int | None = 19, subsample_size: int | None = 1000000, verbose: bool = True) → float[source]

Calibrate the decision threshold in binary classification to optimize a given metric. You can pass the output of this method as input to predictor.set_decision_threshold to update the predictor. Will raise an AssertionError if predictor.problem_type != ‘binary’.

Note that while calibrating the decision threshold can help to improve a given metric, other metrics may end up having worse scores. For example, calibrating on balanced_accuracy will often harm accuracy. Users should keep this in mind while leveraging decision threshold calibration.

Parameters:

  • data (Union[str, pd.DataFrame], default = None) – The data to use for calibration. Must contain the label column. We recommend to keep this value as None unless you are an advanced user and understand the implications. If None, will use internal data such as the holdout validation data or out-of-fold predictions.

  • metric (autogluon.core.metrics.Scorer or str, default = None) – The metric to optimize during calibration. If None, uses predictor.eval_metric.

  • model (str, default = 'best') – The model to use prediction probabilities of when calibrating the threshold. If ‘best’, will use predictor.model_best.

  • decision_thresholds (int | List[float], default = 25) – The number of decision thresholds on either side of 0.5 to search. The default of 25 will result in 51 searched thresholds: [0.00, 0.02, 0.04, …, 0.48, 0.50, 0.52, …, 0.96, 0.98, 1.00] Alternatively, a list of decision thresholds can be passed and only the thresholds in the list will be searched.

  • secondary_decision_thresholds (int | None, default = 19) –

    The number of secondary decision thresholds to check on either side of the threshold identified in the first phase. Skipped if None. For example, if decision_thresholds=50 and 0.14 was identified as the optimal threshold, while secondary_decision_threshold=9,

    Then the following additional thresholds are checked:

    [0.131, 0.132, 0.133, 0.134, 0.135, 0.136, 0.137, 0.138, 0.139, 0.141, 0.142, 0.143, 0.144, 0.145, 0.146, 0.147, 0.148, 0.149]

  • subsample_size (int | None, default = 1000000) – When subsample_size is not None and data contains more rows than subsample_size, samples to subsample_size rows to speed up calibration. Usually it is not necessary to use more than 1 million rows for calibration.

  • verbose (bool, default = True) – If True, will log information about the calibration process.

Returns:

  • Decision Threshold (A float between 0 and 1 defining the decision boundary for predictions that)

  • maximizes the metric score on the data for the model.