Article

Mining distance-based outliers in near linear time with randomization and a simple pruning rule

Authors:

Stephen D. Bay,

Mark SchwabacherAuthors Info & Claims

KDD '03: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 29 - 38

https://doi.org/10.1145/956750.956758

Published: 24 August 2003 Publication History

Abstract

Defining outliers by their distance to neighboring examples is a popular approach to finding unusual examples in a data set. Recently, much work has been conducted with the goal of finding fast algorithms for this task. We show that a simple nested loop algorithm that in the worst case is quadratic can give near linear time performance when the data is in random order and a simple pruning rule is used. We test our algorithm on real high-dimensional data sets with millions of examples and show that the near linear scaling holds over several orders of magnitude. Our average case analysis suggests that much of the efficiency is because the time to process non-outliers, which are the majority of examples, does not depend on the size of the data set.

References

[1]

C. C. Aggarwal and P. S. Yu. Outlier detection for high dimensional data. In Proceedings of the ACM SIGMOD International Conference on Management of Data, 2001.

Digital Library

[2]

F. Angiulli and C. Pizzuti. Fast outlier detection in high dimensional spaces. In Proceedings of the Sixth European Conference on the Principles of Data Mining and Knowledge Discovery, pages 15--26, 2002.

Digital Library

[3]

V. Barnett and T. Lewis. Outliers in Statistical Data. John Wiley & Sons, 1994.

[4]

J. L. Bentley. Multidimensional binary search trees used for associative searching. Communications of the ACM, 18(9):509--517, 1975.

Digital Library

[5]

S. Berchtold, D. Keim, and H.-P. Kreigel. The X-tree: an index structure for high-dimensional data. In Proceedings of the 22nd International Conference on Very Large Databases, pages 28--39, 1996.

Digital Library

[6]

G. Bisson. Learning in FOL with a similarity measure. In Proceedings of the Tenth National Conference on Artificial Intelligence, pages 82--87, 1992.

Digital Library

[7]

R. J. Bolton and D. J. Hand. Statistical fraud detection: A review (with discussion). Statistical Science, 17(3):235--255, 2002.

[8]

M. M. Breunig, H. Kriegel, R. T. Ng, and J. Sander. LOF: Identifying density-based local outliers. In Proceedings of the ACM SIGMOD International Conference on Management of Data, 2000.

Digital Library

[9]

W. Emde and D. Wettschereck. Relational instance-based learning. In Proceedings of the thirteenth International Conference on Machine Learning, 1996.

Digital Library

[10]

E. Eskin, A. Arnold, M. Prerau, L. Portnoy, and S. Stolfo. A geometric framework for unsupervised anomaly detection: Detecting intrusions in unlabeled data. In Data Mining for Security Applications, 2002.

[11]

E. Fix and J. L. Hodges. Discriminatory analysis: Nonparametric discrimination: Small sample performance. Technical Report Project 21-49-004, Report Number 11, USAF School of Aviation Medicine, Randolf Field, Texas, 1952.

[12]

R. Guttmann. A dynamic index structure for spatial searching. In Proceedings of the 1984 ACM SIGMOD International Conference on Management of Data, pages 47--57, 1984.

Digital Library

[13]

D. Hawkins. Identification of outliers. Chapman and Hall, 1980.

[14]

S. Hettich and S. D. Bay. The UCI KDD archive. {http://kdd.ics.uci.edu/}. Irvine, CA: University of California, Department of Information and Computer Science, 1999.

[15]

T. Horvath, S. Wrobel, and U. Bohnebeck. Relational instance-based learning with lists and terms. Machine Learning, 43:53--80, 2001.

Digital Library

[16]

E. M. Knorr and R. T. Ng. Finding intensional knowledge of distance-based outliers. In Proceedings of the 25th VLDB Conference, 1999.

Digital Library

[17]

E. M. Knorr, R. T. Ng, and V. Tucakov. Distance-based outliers: algorithms and applications. VLDB Journal: Very Large Databases, 8(3--4):237--253, 2000.

Digital Library

[18]

T. Lane and C. E. Brodley. Temporal sequence learning and data reduction for anomaly detection. ACM Transactions on Information and System Security, 2(3):295--331, 1999.

Digital Library

[19]

S. Ramaswamy, R. Rastogi, and K. Shim. Efficient algorithms for mining outliers from large data sets. In Proceedings of the ACM SIGMOD Conference, pages 427--438, 2000.

Digital Library

[20]

S. Ruggles and M. Sobek. Integrated public use microdata series: Version 2.0. {http://www.ipums.umn.edu/}, 1997.

[21]

Rulequest Research. Gritbot. {http://www.rulequest.com/}.

Cited By

Hu YYan JCao EYu YTian HHuang H(2024)Approach and Landing Energy Prediction Based on a Long Short-Term Memory ModelAerospace10.3390/aerospace1103022611:3(226)Online publication date: 14-Mar-2024
https://doi.org/10.3390/aerospace11030226
Riveiros AXing YBastianello NJohansson K(2024)Real-Time Anomaly Detection and Categorization for Satellite Reaction Wheels2024 European Control Conference (ECC)10.23919/ECC64448.2024.10591184(253-260)Online publication date: 25-Jun-2024
https://doi.org/10.23919/ECC64448.2024.10591184
Hartmann CSpeth FSabath DSellmaier F(2024)METIS: An AI Assistant Enabling Autonomous Spacecraft Operations for Human Exploration Missions2024 IEEE Aerospace Conference10.1109/AERO58975.2024.10521154(1-22)Online publication date: 2-Mar-2024
https://doi.org/10.1109/AERO58975.2024.10521154
Show More Cited By

Index Terms

Mining distance-based outliers in near linear time with randomization and a simple pruning rule
1. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

Mining Top-n Distance-Based Outliers in Large Datasets with Simple Pruning Rule
ICEICE '12: Proceedings of the 2012 Second International Conference on Electric Information and Control Engineering - Volume 01

In this paper an efficient distance-based algorithm, named RAP (Ranking After Pruning), for detecting top-n outliers in large dataset is presented. The algorithm consists of two phases. In the first phase, the original input data are divided into ...
Robust clustering by pruning outliers

In many applications of C-means clustering, the given data set often contains noisy points. These noisy points will affect the resulting clusters, especially if they are far away from the data points. In this paper, we develop a pruning approach for ...
Detection of local and clustered outliers based on the density–distance decision graph
Abstract
Outlier detection tasks refer to identifying the objects that have different characteristics from the normal observations. Most existing approaches detect outliers from the global perspective, which can effectively detect global ...
Highlights
- A density–distance decision graph based outlier detection method is proposed.
- ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '03: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining

August 2003

736 pages

ISBN:1581137370

DOI:10.1145/956750

Conference Chair:
Lise Getoor
University of Maryland, College Park
,
General Chair:
Ted Senator
DARPA
,
Program Chairs:
Pedro Domingos
University of Washington
,
Christos Faloutsos
Carnegie Mellon University

Copyright © 2003 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

KDD03

Sponsor:

KDD03: The Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 24 - 27, 2003

Washington, D.C.

Acceptance Rates

KDD '03 Paper Acceptance Rate 46 of 298 submissions, 15%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '24

Sponsor:
sigkdd
sigkdd

The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 25 - 29, 2024

Barcelona , Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

306
Total Citations
View Citations
2,238
Total Downloads

Downloads (Last 12 months)66
Downloads (Last 6 weeks)8

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hu YYan JCao EYu YTian HHuang H(2024)Approach and Landing Energy Prediction Based on a Long Short-Term Memory ModelAerospace10.3390/aerospace1103022611:3(226)Online publication date: 14-Mar-2024
https://doi.org/10.3390/aerospace11030226
Riveiros AXing YBastianello NJohansson K(2024)Real-Time Anomaly Detection and Categorization for Satellite Reaction Wheels2024 European Control Conference (ECC)10.23919/ECC64448.2024.10591184(253-260)Online publication date: 25-Jun-2024
https://doi.org/10.23919/ECC64448.2024.10591184
Hartmann CSpeth FSabath DSellmaier F(2024)METIS: An AI Assistant Enabling Autonomous Spacecraft Operations for Human Exploration Missions2024 IEEE Aerospace Conference10.1109/AERO58975.2024.10521154(1-22)Online publication date: 2-Mar-2024
https://doi.org/10.1109/AERO58975.2024.10521154
Guo HZhou ZZhao DGaaloul W(2024)EGNN: Energy-efficient anomaly detection for IoT multivariate time series data using graph neural networkFuture Generation Computer Systems10.1016/j.future.2023.09.028151(45-56)Online publication date: Feb-2024
https://doi.org/10.1016/j.future.2023.09.028
Ghamry FEl-Banby GEl-Fishawy AEl-Samie FDessouky M(2024)A survey of anomaly detection techniquesJournal of Optics10.1007/s12596-023-01147-4Online publication date: 16-Feb-2024
https://doi.org/10.1007/s12596-023-01147-4
Shahbaz Nejad BRoch PHandte MMarrón P(2024)A visual foreign object detection system for wireless charging of electric vehiclesMachine Vision and Applications10.1007/s00138-024-01553-z35:4Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s00138-024-01553-z
Xu HLiang ZHuang KHuang GHe Y(2023)Pre-Cutoff Value Calculation Method for Accelerating Metric Space Outlier DetectionInternational Journal of Grid and High Performance Computing10.4018/IJGHPC.33412516:1(1-17)Online publication date: 28-Nov-2023
https://dl.acm.org/doi/10.4018/IJGHPC.334125
Yang YGang WYuan JZhang ZTian C(2023)Energy Consumption Patterns and Characteristics of College Dormitory Buildings Based on Unsupervised Data Mining MethodBuildings10.3390/buildings1303066613:3(666)Online publication date: 2-Mar-2023
https://doi.org/10.3390/buildings13030666
Yi XHuang PChe S(2023)Application of Knowledge Graph Technology with Integrated Feature Data in Spacecraft Anomaly DetectionApplied Sciences10.3390/app13191090513:19(10905)Online publication date: 30-Sep-2023
https://doi.org/10.3390/app131910905
Gao JLong C(2023)High-Dimensional Approximate Nearest Neighbor Search: with Reliable and Efficient Distance Comparison OperationsProceedings of the ACM on Management of Data10.1145/35892821:2(1-27)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1145/3589282
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents