Article

Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-Out Classifiers

Authors:

Nataraj Jammalamadaka,

Theodore L. WillkeAuthors Info & Claims

Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII

Pages 560 - 574

https://doi.org/10.1007/978-3-030-01237-3_34

Published: 08 September 2018 Publication History

Abstract

As deep learning methods form a critical part in commercially important applications such as autonomous driving and medical diagnostics, it is important to reliably detect out-of-distribution (OOD) inputs while employing these algorithms. In this work, we propose an OOD detection algorithm which comprises of an ensemble of classifiers. We train each classifier in a self-supervised manner by leaving out a random subset of training data as OOD data and the rest as in-distribution (ID) data. We propose a novel margin-based loss over the softmax output which seeks to maintain at least a margin m between the average entropy of the OOD and in-distribution samples. In conjunction with the standard cross-entropy loss, we minimize the novel loss to train an ensemble of classifiers. We also propose a novel method to combine the outputs of the ensemble of classifiers to obtain OOD detection score and class prediction. Overall, our method convincingly outperforms Hendrycks et al. [7] and the current state-of-the-art ODIN [13] on several OOD detection benchmarks.

References

[1]

Bendale, A., Boult, T.E.: Towards open world recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1893–1902 (2015)

[2]

Bendale, A., Boult, T.E.: Towards open set deep networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1563–1572 (2016)

[3]

Chandola V, Banerjee A, and Kumar V Anomaly detection: a survey ACM Comput. Surv. 2009 41 3 15:1-15:58

[4]

Fujimaki, R., Yairi, T., Machida, K.: An approach to spacecraft anomaly detection problem using kernel feature space. In: KDD, pp. 401–410 (2005)

[5]

Goadrich M, Oliphant L, and Shavlik J Creating ensembles of first-order clauses to improve recall-precision curves Mach. Learn. 2006 64 231-262

[6]

Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems (NIPS) (2014)

[7]

Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: International Conference on Learning Representations (ICLR) (2017)

[8]

Huang, G., Liu, Z., Weinberger, K.Q.: Densely connected convolutional networks (2016). arXiv preprint arXiv:1608.06993

[9]

https://tiny-imagenet.herokuapp.com

[10]

Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images

[11]

LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., Huang, F.: A tutorial on energy-based learning. In: Predicting structured data (2006)

[12]

Lee, K., Lee, H., Lee, K., Shin, J.: Training confidence-calibrated classifiers for detecting out-of-distribution samples. In: International Conference on Learning Representations (ICLR) (2018). https://openreview.net/forum?id=ryiAv2xAZ

[13]

Liang, S., Li, Y., Srikant, R.: Enhancing the reliability of out-of-distribution image detection in neural networks. In: International Conference on Learning Representations (ICLR) (2018)

[14]

Liu, F.T., Ting, K.M., Zhou, Z.: Isolation forest. In: ICDM, pp. 413–422 (2008)

[15]

Lu W and Traoré I Unsupervised anomaly detection using an evolutionary extension of k-means algorithm IJICS 2008 2 2 107-139

[16]

Phua C, Alahakoon D, and Lee VCS Minority report in fraud detection: classification of skewed data SIGKDD Explor. 2004 6 1 50-59

[17]

Rudd EM, Jain LP, Scheirer WJ, and Boult TE The extreme value machine IEEE Trans. Pattern Anal. Mach. Intell. 2018 40 3 762-768

[18]

Scheirer WJ, Jain LP, and Boult TE Probability models for open set recognition IEEE Trans. Pattern Anal. Mach. Intell. 2014 36 11 2317-2324

[19]

Scheirer WJ, de Rezende Rocha A, Sapkota A, and Boult TE Toward open set recognition IEEE Trans. Pattern Anal. Mach. Intell. 2013 35 7 1757-1772

[20]

Xu, P., Ehinger, K.A., Zhang, Y., Finkelstein, A., Kulkarni, S.R., Xiao, J.: Turkergaze: crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755 (2015)

[21]

Yu, F., Zhang, Y., Song, S., Seff, A., Xiao, J.: LSUN: construction of a large- scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 (2015)

[22]

Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)

Cited By

Kochanthara SSingh TForrai ACleophas L(2024)Safety of Perception Systems for Automated Driving: A Case Study on ApolloACM Transactions on Software Engineering and Methodology10.1145/363196933:3(1-28)Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1145/3631969
Yatbaz HDianati MWoodman R(2024)Introspection of DNN-Based Perception Functions in Automated Driving Systems: State-of-the-Art and Open Research ChallengesIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.331507025:2(1112-1130)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/TITS.2023.3315070
Wang BMine TFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Optimizing Upstream Representations for Out-of-Domain Detection with Supervised Contrastive LearningProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615001(2585-2595)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615001
Show More Cited By

Index Terms

Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-Out Classifiers
1. Computing methodologies
  1. Machine learning
2. Software and its engineering

Index terms have been assigned to the content through auto-classification.

Recommendations

Improved classification with allocation method and multiple classifiers

We propose a new allocation method for building a classification ensemble.Allocation method uses multiple classifiers: the allocator and micro classifiers.Allocator separates the dataset and allocates them to one of micro classifiers.Allocator is based ...
Self-supervised out-of-distribution detection in wireless capsule endoscopy images
Abstract
While deep learning has displayed excellent performance in a broad spectrum of application areas, neural networks still struggle to recognize what they have not seen, i.e., out-of-distribution (OOD) inputs. In the medical field, building robust ...
Graphical abstract

Display Omitted
Highlights
- We introduce a new self-supervised out-of-distribution for Wireless Capsule Endoscopy Images.
- The proposed model generates patch embeddings as a self-supervised task.
- The proposed approach show an improvement compared to state-of-...
Autoencoding Binary Classifiers for Supervised Anomaly Detection
PRICAI 2019: Trends in Artificial Intelligence
Abstract
We propose the Autoencoding Binary Classifiers (ABC), a novel supervised anomaly detector based on the Autoencoder (AE). There are two main approaches in anomaly detection: supervised and unsupervised. The supervised approach accurately detects ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII

Sep 2018

845 pages

ISBN:978-3-030-01236-6

DOI:10.1007/978-3-030-01237-3

Editors:
Vittorio Ferrari
Google Research, Zurich, Switzerland
,
Martial Hebert
Carnegie Mellon University, Pittsburgh, PA, USA
,
Cristian Sminchisescu
Google Research, Zurich, Switzerland
,
Yair Weiss
Hebrew University of Jerusalem, Jerusalem, Israel

© Springer Nature Switzerland AG 2018.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 08 September 2018

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kochanthara SSingh TForrai ACleophas L(2024)Safety of Perception Systems for Automated Driving: A Case Study on ApolloACM Transactions on Software Engineering and Methodology10.1145/363196933:3(1-28)Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1145/3631969
Yatbaz HDianati MWoodman R(2024)Introspection of DNN-Based Perception Functions in Automated Driving Systems: State-of-the-Art and Open Research ChallengesIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.331507025:2(1112-1130)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/TITS.2023.3315070
Wang BMine TFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Optimizing Upstream Representations for Out-of-Domain Detection with Supervised Contrastive LearningProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615001(2585-2595)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615001
Yang THuang YXie YLiu JWang S(2023)MixOOD: Improving Out-of-distribution Detection with Enhanced Data MixupACM Transactions on Multimedia Computing, Communications, and Applications10.1145/357893519:5(1-18)Online publication date: 16-Mar-2023
https://dl.acm.org/doi/10.1145/3578935
Huang WWang HXia JWang CZhang JKoyejo SMohamed SAgarwal ABelgrave DCho KOh A(2022)Density-driven regularization for out-of-distribution detectionProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3600335(887-900)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.5555/3600270.3600335
Mohseni SWang HXiao CYu ZWang ZYadawa J(2022)Taxonomy of Machine Learning Safety: A Survey and PrimerACM Computing Surveys10.1145/355138555:8(1-38)Online publication date: 23-Dec-2022
https://dl.acm.org/doi/10.1145/3551385
Shan SDing WWenger EZheng HZhao BYin HStavrou ACremers CShi E(2022)Post-breach RecoveryProceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security10.1145/3548606.3560561(2611-2625)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3548606.3560561
Song YWang DZhang ARangwala H(2022)Learning on Graphs with Out-of-Distribution NodesProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3534678.3539457(1635-1645)Online publication date: 14-Aug-2022
https://dl.acm.org/doi/10.1145/3534678.3539457
Wang BMine THong JBures MPark JCerny T(2022)Practical and efficient out-of-domain detection with adversarial learningProceedings of the 37th ACM/SIGAPP Symposium on Applied Computing10.1145/3477314.3507089(853-862)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3477314.3507089
Li RZhang CZhou HShi CLuo Y(2022)Out-of-Distribution Identification: Let Detector Tell Which I Am Not SureComputer Vision – ECCV 202210.1007/978-3-031-20080-9_37(638-654)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-20080-9_37
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents