Version 0.4.0

We’re happy to announce the AutoGluon 0.4 release. 0.4 contains major enhancements to Tabular and Text modules, along with many quality of life improvements and fixes.

This release is non-breaking when upgrading from v0.3.1. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.

This release contains 151 commits from 14 contributors!

See the full commit change-log here: https://github.com/autogluon/autogluon/compare/v0.3.1…v0.4.0

Special thanks to @zhiqiangdon, @willsmithorg, @DolanTheMFWizard, @truebluejason, @killerSwitch, and @Xilorole who were first time contributors to AutoGluon this release!

Full Contributor List (ordered by # of commits):

  • @Innixma, @yinweisu, @gradientsky, @zhiqiangdon, @jwmueller, @willsmithorg, @sxjscience, @DolanTheMFWizard, @truebluejason, @taesup-aws, @Xilorole, @mseeger, @killerSwitch, @rschmucker

This version supports Python versions 3.7 to 3.9.

Changes

General

  • AutoGluon now supports Windows OS! Both CPU and GPU are supported on Windows.

  • AutoGluon now supports Python 3.9. Python 3.6 is no longer supported.

  • AutoGluon has migrated from MXNet to PyTorch for all deep learning models resulting in major speedups.

  • New tutorials showcasing cloud training and deployment with AWS SageMaker and Lambda.

Text

AutoGluon-Text is refactored with PyTorch Lightning. It now supports backbones in huggingface/transformers. The new version has better performance, faster training time, and faster inference speed. In addition, AutoGluon-Text now supports solving multilingual problems and a new AutoMMPredictor has been implemented for automatically building multimodal DL models.

  • Better Performance

  • Faster Speed

    • The new version has ~2.88x speedup in training and ~1.40x speedup in inference. With g4dn.12x instance, the model can achieve an additional 2.26x speedup with 4 GPUs.

  • Multilingual Support

    • AutoGluon-Text now supports solving multilingual problems via cross-lingual transfer (Tutorial). This is triggered by setting presets="multilingual". You can now train a model on the English dataset and directly apply the model on datasets in other languages such as German, Japanese, Italian, etc.

  • AutoMMPredictor for Multimodal Problems

    • Support an experimental AutoMMPredictor that supports fusion image backbones in timm, text backbone in huggingface/transformers, and multimodal backbones like CLIP (Tutorial). It may perform better than ensembling ImagePredictor + TextPredictor.

  • Other Features

    • Support continuous training from an existing checkpoint. You may just call .fit() again after a previous trained model has been loaded.

Thanks to @zhiqiangdon and @sxjscience for contributing the AutoGluon-Text refactors! (#1537, #1547, #1557, #1565, #1571, #1574, #1578, #1579, #1581, #1585, #1586)

Tabular

AutoGluon-Tabular has been majorly enhanced by numerous optimizations in 0.4. In summation, these improvements have led to a:

  • ~2x training speedup in Good, High, and Best quality presets.

  • ~1.3x inference speedup.

  • 63% win-rate vs AutoGluon 0.3.1 (Results from AutoMLBenchmark)

    • 93% win-rate vs AutoGluon 0.3.1 on datasets with >=100,000 rows of data (!!!)

Specific updates:

  • Added infer_limit and infer_limit_batch_size as new fit-time constraints (Tutorial). This allows users to specify the desired end-to-end inference latency of the final model and AutoGluon will automatically train models to satisfy the constraint. This is extremely useful for online-inference scenarios where you need to satisfy an end-to-end latency constraint (for example 50ms). @Innixma (#1541, #1584)

  • Implemented automated semi-supervised and transductive learning in TabularPredictor. Try it out via TabularPredictor.fit_pseudolabel(...)! @DolanTheMFWizard (#1323, #1382)

  • Implemented automated feature pruning (i.e. feature selection) in TabularPredictor. Try it out via TabularPredictor.fit(..., feature_prune_kwargs={})! @truebluejason (#1274, #1305)

  • Implemented automated model calibration to improve AutoGluon’s predicted probabilities for classification problems. This is enabled by default, and can be toggled via the calibrate fit argument. @DolanTheMFWizard (#1336, #1374, #1502)

  • Implemented parallel bag training via Ray. This results in a ~2x training speedup when bagging is enabled compared to v0.3.1 with the same hardware due to more efficient usage of resources for models that cannot effectively use all cores. @yinweisu (#1329, #1415, #1417, #1423)

  • Added adaptive early stopping logic which greatly improves the quality of models within a time budget. @Innixma (#1380)

  • Added automated model calibration in quantile regression. @taesup-aws (#1388)

  • Enhanced datetime feature handling. @willsmithorg (#1446)

  • Added support for custom confidence levels in feature importance. @jwmueller (#1328)

  • Improved neural network HPO search spaces. @jwmueller (#1346)

  • Optimized one-hot encoding preprocessing. @Innixma (#1376)

  • Refactored refit_full logic to majorly simplify user model contributions and improve multimodal support with advanced presets. @Innixma (#1567)

  • Added experimental TabularPredictor config helper. @gradientsky (#1491)

  • New Tutorials

Tabular Models

NEW: TabularNeuralNetTorchModel (alias: ‘NN_TORCH’)

As part of the migration from MXNet to Torch, we have created a Torch based counterpart to the prior MXNet tabular neural network model. This model has several major advantages, such as:

  • 1.9x faster training speed

  • 4.7x faster inference speed

  • 51% win-rate vs MXNet Tabular NN

This model has replaced the MXNet tabular neural network model in the default hyperparameters configuration, and is enabled by default.

Thanks to @jwmueller and @Innixma for contributing TabularNeuralNetTorchModel to AutoGluon! (#1489)

NEW: VowpalWabbitModel (alias: ‘VW’)

VowpalWabbit has been added as a new model in AutoGluon. VowpalWabbit is not installed by default, and must be installed separately. VowpalWabbit is used in the hyperparameters='multimodal' preset, and the model is a great option to use for datasets containing text features.

To install VowpalWabbit, specify it via pip install autogluon.tabular[all, vowpalwabbit] or pip install "vowpalwabbit>=8.10,<8.11"

Thanks to @killerSwitch for contributing VowpalWabbitModel to AutoGluon! (#1422)

XGBoostModel (alias: ‘XGB’)

  • Optimized model serialization method, which results in 5.5x faster inference speed and halved disk usage. @Innixma (#1509)

  • Adaptive early stopping logic leading to 54.7% win-rate vs prior implementation. @Innixma (#1380)

  • Optimized training speed with expensive metrics such as F1 by ~10x. @Innixma (#1344)

  • Optimized num_cpus default to equal physical cores rather than virtual cores. @Innixma (#1467)

CatBoostModel (alias: ‘CAT’)

  • CatBoost now incorporates callbacks which make it more stable and resilient to memory errors, along with more advanced adaptive early stopping logic that leads to 63.2% win-rate vs prior implementation. @Innixma (#1352, #1380)

LightGBMModel (alias: ‘GBM’)

  • Optimized training speed with expensive metrics such as F1 by ~10x. @Innixma (#1344)

  • Adaptive early stopping logic leading to 51.1% win-rate vs prior implementation. @Innixma (#1380)

  • Optimized num_cpus default to equal physical cores rather than virtual cores. @Innixma (#1467)

FastAIModel (alias: ‘FASTAI’)

  • Added adaptive batch size selection and epoch selection. @gradientsky (#1409)

  • Enabled HPO support in FastAI (previously HPO was not supported for FastAI). @Innixma (#1408)

  • Made FastAI training deterministic (it is now consistently seeded). @Innixma (#1419)

  • Fixed GPU specification in FastAI to respect the num_gpus parameter. @Innixma (#1421)

  • Forced correct number of threads during fit and inference to avoid issues with global thread updates. @yinweisu (#1535)

LinearModel (alias: ‘LR’)

Linear models have been accelerated by 20x in training and 20x in inference thanks to a variety of optimizations. To get the accelerated training speeds, please install scikit-learn-intelex via pip install "scikit-learn-intelex>=2021.5,<2021.6"

Note that currently LinearModel is not enabled by default in AutoGluon, and must be specified in hyperparameters via the key 'LR'. Further testing is planned to incorporate LinearModel as a default model in future releases.

Thanks to the scikit-learn-intelex team and @Innixma for the LinearModel optimizations! (#1378)

Vision

  • Refactored backend logic to be more robust. @yinweisu (#1427)

  • Added support for inference via CPU. Previously, inferring without GPU would error. @yinweisu (#1533)

  • Refactored HPO logic. @Innixma (#1511)

Miscellaneous

  • AutoGluon no longer depends on ConfigSpace, cython, dill, paramiko, autograd, openml, d8, and graphviz. This greatly simplifies installation of AutoGluon, particularly on Windows.

  • Entirely refactored HPO logic to break dependencies on ConfigSpace and improve stability and ease of development. @Innixma HPO has been simplified to use random search in this release while we work on re-introducing the more advanced HPO methods such as bayesopt in a future release. Additionally, removed 40,000 lines of out-dated code to streamline future development. @Innixma (#1397, #1411, #1414, #1431, #1443, #1511)

  • Added autogluon.common to simplify dependency management for future submodules. @Innixma (#1386)

  • Removed autogluon.mxnet and autogluon.extra submodules as part of code cleanup. @Innixma (#1397, #1411, #1414)

  • Refactored logging to avoid interfering with other packages. @yinweisu (#1403)

  • Fixed logging output on Kaggle, previously no logs would be displayed while fitting AutoGluon in a Kaggle kernel. @Innixma (#1468)

  • Added platform tests for Linux, MacOS, and Windows. @yinweisu (#1464, #1506, #1513)

  • Added ROADMAP.md to highlight past, present, and future feature prioritization and progress to the community. @Innixma (#1420)

  • Various documentation and CI improvements

    • @jwmueller (#1379, #1408, #1429)

    • @gradientsky (#1383, #1387, #1471, #1500)

    • @yinweisu (#1441, #1482, #1566, #1580)

    • @willsmithorg (#1476, #1483)

    • @Xilorole (#1526)

    • @Innixma (#1452, #1453, #1528, #1577, #1584, #1588, #1593)

  • Various backend enhancements / refactoring / cleanup

    • @DolanTheMFWizard (#1319)

    • @gradientsky (#1320, #1366, #1385, #1448, #1488, #1490, #1570, #1576)

    • @mseeger (#1349)

    • @yinweisu (#1497, #1503, #1512, #1563, #1573)

    • @willsmithorg (#1525, #1543)

    • @Innixma (#1311, #1313, #1327, #1331, #1338, #1345, #1369, #1377, #1380, #1408, #1410, #1412, #1419, #1425, #1428, #1462, #1465, #1562, #1569, #1591, #1593)

  • Various bug fixes

    • @jwmueller (#1314, #1356)

    • @yinweisu (#1472, #1499, #1504, #1508, #1516)

    • @gradientsky (#1514)

    • @Innixma (#1304, #1325, #1326, #1337, #1365, #1395, #1405, #1587, #1599)