Article

HOT: hypergraph-based outlier test for categorical data

Authors:

Jeffrey X. YuAuthors Info & Claims

PAKDD '03: Proceedings of the 7th Pacific-Asia conference on Advances in knowledge discovery and data mining

Pages 399 - 410

Published: 30 April 2003 Publication History

Abstract

As a widely used data mining technique, outlier detection is a process which aims at finding anomalies with good explanations. Most existing methods are designed for numeric data. They will have problems with real-life applications that contain categorical data. In this paper, we introduce a novel outlier mining method based on a hypergraph model. Since hypergraphs precisely capture the distribution characteristics in data subspaces, this method is effective in identifying anomalies in dense subspaces and presents good interpretations for the local outlierness. By selecting the most relevant subspaces, the problem of "curse of dimensionality" in very large databases can also be ameliorated. Furthermore, the connectivity property is used to replace the distance metrics, so that the distance-based computation is not needed anymore, which enhances the robustness for handling missing-value data. The fact, that connectivity computation facilitates the aggregation operations supported by most SQL-compatible database systems, makes the mining process much efficient. Finally, experiments and analysis show that our method can find outliers in categorical data with good performance and quality.

References

[1]

C. Aggarwal and P. Yu. Outlier detection for high dimensional data. In Proc. of SIGMOD'2001, pages 37-47, 2001.

Digital Library

[2]

R. Agrawal and R. Srikant. Fast algorithms for mining association rules in large databases. In Proc. of VLDB'94, pages 487-499, 1994.

Digital Library

[3]

V. Barnett and T. Lewis. Outliers In Statistical Data. John Wiley, Reading, New York, 1994.

[4]

M. Breunig, H.-P. Kriegel, R. Ng, and J. Sander. Optics-of: Identifying local outliers. In Proc. of PKDD'99, pages 262-270, 1999.

Digital Library

[5]

M. Breunig, H.-P. Kriegel, R. Ng, and J. Sander. Lof: Identifying density-based local outliers. In Proc. of SIGMOD'2000, pages 93-104, 2000.

Digital Library

[6]

M. Ester, H.-P. Kriegel, J. Sander, and X. Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proc. of KDD'96, pages 226-231, 1996.

[7]

S. Guha, R. Rastogi, and K. Shim. Cure: An efficient clustering algorithm for large databases. In Proc. of SIGMOD'98, pages 73-84, 1998.

Digital Library

[8]

D. Hawkins. Identification of Outliers. Chapman and Hall, Reading, London, 1980.

[9]

F. Hussain, H. Liu, C. L. Tan, and M. Dash. Discretization: An enabling technique. Technical Report TRC6/99, National University of Singapore, School of Computing, 1999.

[10]

W. Jin, A. K. Tung, and J. Han. Mining top-n local outliers in large databases. In Proc. of KDD'2001, pages 293-298, 2001.

Digital Library

[11]

E. Knorr and R. Ng. Algorithms for mining distance-based outliers in large datasets. In Proc. of VLDB'98, pages 392-403, 1998.

Digital Library

[12]

E. Knorr and R. Ng. Finding intensional knowledge of distance-based outliers. In Proc. of VLDB'99, pages 211-222, 1999.

Digital Library

[13]

G. Merz and P. Murphy. Uci repository of machine learning databases. Technical Report, University of California, Department of Information and Computer Science: http://www.ics.uci.edu/mlearn/MLRepository.html, 1996.

[14]

F. Preparata and M. Shamos. Computational Geometry: an Introduction. Springer-Verlag, Reading, New York, 1988.

Digital Library

[15]

S. Ramaswamy, R. Rastogi, and K. Shim. Efficient algorithms for mining outliers from large data sets. In Proc. of SIGMOD'2000, pages 427-438, 2000.

Digital Library

[16]

I. Ruts and P. Rousseeuw. Computing depth contours of bivariate point clouds. Journal of Computational Statistics and data Analysis, 23:153-168, 1996.

Digital Library

[17]

S. Shekhar, C.-T. Lu, and P. Zhang. Detecting graph-based spatial outliers: Algorithms and applications (a summary of results). In Proc. of KDD'2001, 2001.

Digital Library

Cited By

Taha AHadi A(2019)Anomaly Detection Methods for Categorical DataACM Computing Surveys10.1145/331273952:2(1-35)Online publication date: 30-May-2019
https://dl.acm.org/doi/10.1145/3312739
Yoon DNiu NMozafari BÖzcan FKoutrika GMadden S(2016)DBSherlockProceedings of the 2016 International Conference on Management of Data10.1145/2882903.2915218(1599-1614)Online publication date: 26-Jun-2016
https://dl.acm.org/doi/10.1145/2882903.2915218
Sweetlin Hemalatha CVaidehi VLakshmi R(2015)Minimal infrequent pattern based approach for mining outliers in data streamsExpert Systems with Applications: An International Journal10.1016/j.eswa.2014.09.05342:4(1998-2012)Online publication date: 1-Mar-2015
https://dl.acm.org/doi/10.1016/j.eswa.2014.09.053
Show More Cited By

Recommendations

What's hot and what's not: tracking most frequent items dynamically
PODS '03: Proceedings of the twenty-second ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

Most database management systems maintain statistics on the underlying relation. One of the important statistics is that of the "hot items" in the relation: those that appear many times (most frequently, or more than some threshold). For example, end-...
Efficient automatic discovery of 'hot' itemsets

In real life applications the dominant model of the single support, which assumed all itemsets to be of the same nature and importance proved defective. The non-homogeneity of the itemsets on one hand and the non-uniformity of their number of ...
Evolutionary Hot Spots Data Mining - An Architecture for Exploring for Interesting Discoveries
PAKDD '99: Proceedings of the Third Pacific-Asia Conference on Methodologies for Knowledge Discovery and Data Mining

Data Mining delivers novel and useful knowledge from very large collections of data. The task is often characterised as identifying key areas within a very large dataset which have some importance or are otherwise interesting to the data owners. We call ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

PAKDD '03: Proceedings of the 7th Pacific-Asia conference on Advances in knowledge discovery and data mining

April 2003

610 pages

ISBN:3540047603

Editors:
Kyu-Young Wang
Korea Advanced Institute of Science and Technology, Computer Science Department, Daejeon, Korea
,
Jongwoo Jeon
Seoul National University, Department of Statistics, Seoul, Korea
,
Kyuseok Shim
Seoul National University, School of Electrical Engineering and Computer Science, Seoul, Korea
,
Jaideep Srivastava
University of Minnesota, Department of Computer Science and Engineering, Minneapolis, MN

Sponsors

Statistical Research Center for Complex Systems
KAIST: Korea Advanced Institute of Science and Technology
The Korean Datamining Society
Advanced Information Technology Research Center
Korea Info Sci Society: Korea Information Science Society
Air Force Office of Scientific Research/Asian Office of Aerospace R&D

In-Cooperation

SIGKDD: ACM Special Interest Group on Knowledge Discovery in Data

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 30 April 2003

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Taha AHadi A(2019)Anomaly Detection Methods for Categorical DataACM Computing Surveys10.1145/331273952:2(1-35)Online publication date: 30-May-2019
https://dl.acm.org/doi/10.1145/3312739
Yoon DNiu NMozafari BÖzcan FKoutrika GMadden S(2016)DBSherlockProceedings of the 2016 International Conference on Management of Data10.1145/2882903.2915218(1599-1614)Online publication date: 26-Jun-2016
https://dl.acm.org/doi/10.1145/2882903.2915218
Sweetlin Hemalatha CVaidehi VLakshmi R(2015)Minimal infrequent pattern based approach for mining outliers in data streamsExpert Systems with Applications: An International Journal10.1016/j.eswa.2014.09.05342:4(1998-2012)Online publication date: 1-Mar-2015
https://dl.acm.org/doi/10.1016/j.eswa.2014.09.053
Wu SWang S(2011)Parameter-free anomaly detection for categorical dataProceedings of the 7th international conference on Machine learning and data mining in pattern recognition10.5555/2033831.2033841(112-126)Online publication date: 30-Aug-2011
https://dl.acm.org/doi/10.5555/2033831.2033841
Chandola VBanerjee AKumar V(2009)Anomaly detectionACM Computing Surveys10.1145/1541880.154188241:3(1-58)Online publication date: 30-Jul-2009
https://dl.acm.org/doi/10.1145/1541880.1541882
Angiulli FFassetti FPalopoli L(2009)Detecting outlying properties of exceptional objectsACM Transactions on Database Systems10.1145/1508857.150886434:1(1-62)Online publication date: 23-Apr-2009
https://dl.acm.org/doi/10.1145/1508857.1508864
Ye MLi XOrlowska M(2009)Projected outlier detection in high-dimensional mixed-attributes data setExpert Systems with Applications: An International Journal10.1016/j.eswa.2008.08.03036:3(7104-7113)Online publication date: 1-Apr-2009
https://dl.acm.org/doi/10.1016/j.eswa.2008.08.030
He ZDeng SXu X(2005)An optimization model for outlier detection in categorical dataProceedings of the 2005 international conference on Advances in Intelligent Computing - Volume Part I10.1007/11538059_42(400-409)Online publication date: 23-Aug-2005
https://dl.acm.org/doi/10.1007/11538059_42

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

Media

Figures

Other

Tables

View Table of Contents