tutorial

Statistically sound pattern discovery

Authors:

Wilhelmiina Hämäläinen,

Geoffrey I. WebbAuthors Info & Claims

KDD '14: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining

Page 1976

https://doi.org/10.1145/2623330.2630812

Published: 24 August 2014 Publication History

Get Access

Abstract

Pattern discovery is a core data mining activity. Initial approaches were dominated by the frequent pattern discovery paradigm -- only patterns that occur frequently in the data were explored. Having been thoroughly researched and its limitations now well understood, this paradigm is giving way to a new one, which can be called statistically sound pattern discovery. In this paradigm, the main impetus is to discover statistically significant patterns, which are unlikely to have occurred by chance and are likely to hold in future data. Thus, the new paradigm provides a strict control over false discoveries and overfitting.

This tutorial covers both classic and cutting-edge research topics on pattern discovery combined to statistical significance testing. We start with an advanced introduction to the relevant forms of statistical significance testing, including different schools and alternative models, their underlying assumptions, practical issues, and limitations. We then discuss their application to data mining specific problems, including evaluation of nested patterns, the multiple testing problem, algorithmic strategies and real-world considerations. We present the current state-of-the art solutions and explore in detail how this approach to pattern discovery can deliver efficient and effective discovery of small sets of interesting patterns.

Supplementary Material

Part 1 of 3 (p1976-sidebyside1.mp4)

Download
884.50 MB

Part 2 of 3 (p1976-sidebyside2.mp4)

Download
1188.40 MB

Part 3 of 3 (p1976-sidebyside3.mp4)

Download
1108.36 MB

Cited By

View all

Hämäläinen WWebb G(2019)A tutorial on statistically sound pattern discoveryData Mining and Knowledge Discovery10.1007/s10618-018-0590-x33:2(325-377)Online publication date: 1-Mar-2019
https://dl.acm.org/doi/10.1007/s10618-018-0590-x
Ozaki T(2019)Evaluation Measures for Extended Association Rules Based on Distributed RepresentationsPrimate Life Histories, Sex Roles, and Adaptability10.1007/978-3-030-15035-8_29(305-313)Online publication date: 15-Mar-2019
https://doi.org/10.1007/978-3-030-15035-8_29
Ozaki T(2018)Evaluation Measures for Frequent Itemsets Based on Distributed Representations2018 Sixth International Symposium on Computing and Networking (CANDAR)10.1109/CANDAR.2018.00028(153-159)Online publication date: Nov-2018
https://doi.org/10.1109/CANDAR.2018.00028
Show More Cited By

Index Terms

Statistically sound pattern discovery
1. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

A tutorial on statistically sound pattern discovery

Statistically sound pattern discovery harnesses the rigour of statistical hypothesis testing to overcome many of the issues that have hampered standard data mining approaches to pattern discovery. Most importantly, application of appropriate statistical ...
A novel Boolean algebraic framework for association and pattern mining

Data mining has been defined as the non- trivial extraction of implicit, previously unknown and potentially useful information from data. Association mining and sequential mining analysis are considered as crucial components of strategic control over a ...
Efficient Discovery of the Most Interesting Associations

Self-sufficient itemsets have been proposed as an effective approach to summarizing the key associations in data. However, their computation appears highly demanding, as assessing whether an itemset is self-sufficient requires consideration of all ...

Comments

Information & Contributors

Information

Published In

KDD '14: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining

August 2014

2028 pages

ISBN:9781450329569

DOI:10.1145/2623330

General Chairs:
Sofus Macskassy
Facebook
,
Claudia Perlich
Dstillery
,
Program Chairs:
Jure Leskovec
Stanford University
,
Wei Wang
UCLA
,
Rayid Ghani
University of Chicago

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2014

Check for updates

Author Tags

Qualifiers

Tutorial

Conference

KDD '14

Sponsor:

KDD '14: The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 24 - 27, 2014

New York, New York, USA

Acceptance Rates

KDD '14 Paper Acceptance Rate 151 of 1,036 submissions, 15%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
389
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Hämäläinen WWebb G(2019)A tutorial on statistically sound pattern discoveryData Mining and Knowledge Discovery10.1007/s10618-018-0590-x33:2(325-377)Online publication date: 1-Mar-2019
https://dl.acm.org/doi/10.1007/s10618-018-0590-x
Ozaki T(2019)Evaluation Measures for Extended Association Rules Based on Distributed RepresentationsPrimate Life Histories, Sex Roles, and Adaptability10.1007/978-3-030-15035-8_29(305-313)Online publication date: 15-Mar-2019
https://doi.org/10.1007/978-3-030-15035-8_29
Ozaki T(2018)Evaluation Measures for Frequent Itemsets Based on Distributed Representations2018 Sixth International Symposium on Computing and Networking (CANDAR)10.1109/CANDAR.2018.00028(153-159)Online publication date: Nov-2018
https://doi.org/10.1109/CANDAR.2018.00028
Kawanobe SOzaki T(2017)Extraction of Characteristic Frequent Visual Patterns by Distributed Representation2017 31st International Conference on Advanced Information Networking and Applications Workshops (WAINA)10.1109/WAINA.2017.71(525-530)Online publication date: Mar-2017
https://doi.org/10.1109/WAINA.2017.71

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

A tutorial on statistically sound pattern discovery

A novel Boolean algebraic framework for association and pattern mining

Efficient Discovery of the Most Interesting Associations