Article

Survey of Improving Naive Bayes for Classification

Authors:

Liangxiao Jiang,

Xuesong YanAuthors Info & Claims

ADMA '07: Proceedings of the 3rd international conference on Advanced Data Mining and Applications

Pages 134 - 145

https://doi.org/10.1007/978-3-540-73871-8_14

Published: 06 August 2007 Publication History

Abstract

The attribute conditional independence assumption of naive Bayes essentially ignores attribute dependencies and is often violated. On the other hand, although a Bayesian network can represent arbitrary attribute dependencies, learning an optimal Bayesian network classifier from data is intractable. Thus, learning improved naive Bayes has attracted much attention from researchers and presented many effective and efficient improved algorithms. In this paper, we review some of these improved algorithms and single out four main improved approaches: 1) Feature selection; 2) Structure extension; 3) Local learning; 4) Data expansion. We experimentally tested these approaches using the whole 36 UCI data sets selected by Weka, and compared them to naive Bayes. The experimental results show that all these approaches are effective. In the end, we discuss some main directions for future research on Bayesian network classifiers.

References

[1]

Langley, P., Iba, W., Thomas, K.: An analysis of Bayesian classifiers. In: Proceedings of the Tenth National Conference of Artificial Intelligence, pp. 223-228. AAAI Press, Stanford (1992).

[2]

Pearl, J.: Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, San Francisco, CA (1988).

[3]

Chickering, D.M.: Learning Bayesian networks is NP-Complete. In: Fisher, D., Lenz, H. (eds.) Learning from Data: Artificial Intelligence and Statistics V, pp. 121-130. Springer, Heidelberg (1996).

[4]

Langley, P., Sage, S.: Induction of selective Bayesian classifiers. In: Proceedings of the Tenth Conference on Uncertainty in Artificial Intelligence, pp. 339-406 (1994).

[5]

Jiang, L., Zhang, H., Cai, Z., Su, J.: Evolutional Naive Bayes. In: Proceedings of the 1st International Symposium on Intelligent Computation and its Applications, ISICA, China University of Geosciences Press, pp. 344-350 (2005).

[6]

Kohavi, R., John, G.: Wrappers for Feature Subset Selection. Artificial Intelligence journal, special issue on relevance 97(1-2), 273-324 (1997).

[7]

Ratanamahatana, C.A., Gunopulos, D.: Scaling up the Naive Bayesian Classifier: Using Decision Trees for Feature Selection. In: proceedings of Workshop on Data Cleaning and Preprocessing (DCAP 2002), at IEEE International Conference on Data Mining (ICDM 2002), Maebashi, Japan (2002).

[8]

Friedman, Geiger, Goldszmidt.: Bayesian Network Classifiers. Machine Learning 29, 131-163 (1997).

[9]

Chow, C.K., Liu, C.N.: Approximating discrete probability distributions with dependence trees. IEEE Trans. on Information Theory 14, 462C-467 (1968).

[10]

Keogh, E., Pazzani, M.: Learning augmented Bayesian classifiers: A comparison of distribution-based and classification-based approaches. In: Proceedings of the International Workshop on Artificial Intelligence and Statistics, pp. 225C-230 (1999).

[11]

Zhang, H., Ling, C.X.: An improved learning algorithm for augmented naive Bayes. In: Cheung, D., Williams, G.J., Li, Q. (eds.) PAKDD 2001. LNCS (LNAI), vol. 2035, pp. 581-586. Springer, Heidelberg (2001).

[12]

Jiang, L., Zhang, H., Cai, Z., Su, J.: One Dependence Augmented Naive Bayes. In: Li, X., Wang, S., Dong, Z.Y. (eds.) ADMA 2005. LNCS (LNAI), vol. 3584, pp. 186-194. Springer, Heidelberg (2005).

[13]

Webb, G.I., Boughton, J., Wang, Z.: Not so naive bayes: Aggregating one-dependence estimators. Machine Learning 58, 5-24 (2005).

[14]

Jiang, L., Zhang, H.: Weightily Averaged One-Dependence Estimators. In: Yang, Q., Webb, G. (eds.) PRICAI 2006. LNCS (LNAI), vol. 4099, pp. 970-974. Springer, Heidelberg (2006).

[15]

Zhang, H., Jiang, L., Su, J.: Hidden Naive Bayes. In: Proceedings of the 20th National Conference on Artificial Intelligence, AAAI 2005, pp. 919-924. AAAI Press, Stanford (2005).

[16]

Kohavi, R.: Scaling Up the Accuracy of Naive-Bayes Classifiers: A Decision-Tree Hybrid. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD-96), pp. 202-207. AAAI Press, Stanford (1996).

[17]

Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, CA (1993).

[18]

Frank, E., Hall, M., Pfahringer, B.: Locally Weighted Naive Bayes. In: Proceedings of the Conference on Uncertainty in Artificial Intelligence (2003), pp. 249-256. Morgan Kaufmann, Seattle (2003).

[19]

Jiang, L., Zhang, H., Su, J.: Instance Cloning Local Naive Bayes. In: Kégl, B., Lapalme, G. (eds.) Canadian AI 2005. LNCS (LNAI), vol. 3501, pp. 280-291. Springer, Heidelberg (2005).

[20]

Zheng, Z., Webb, G.I.: Lazy Learning of Bayesian Rules. Machine Learning 41(1), 53-84 (2000).

[21]

Xie, Z., Hsu, W., Liu, Z., Lee, M.: A Selective Neighborhood Based Naive Bayes for Lazy Learning. In: Chen, M.-S., Yu, P.S., Liu, B. (eds.) PAKDD 2002. LNCS (LNAI), vol. 2336, pp. 104-114. Springer, Heidelberg (2002).

[22]

Jiang, L., Zhang, H., Cai, Z.: Dynamic K-Nearest-Neighbor Naive Bayes with Attribute Weighted. In: Wang, L., Jiao, L., Shi, G., Li, X., Liu, J. (eds.) FSKD 2006. LNCS (LNAI), vol. 4223, pp. 365-368. Springer, Heidelberg (2006).

[23]

Jiang, L., Guo, Y.: Learning Lazy Naive Bayesian Classifiers for Ranking. In: Proceedings of the 17th IEEE International Conference on Tools with Artificial Intelligence, ICTAI 2005, pp. 412-416. IEEE Computer Society Press, Los Alamitos (2005).

[24]

Jiang, L., Zhang, H.: Learning Instance Greedily Cloning Naive Bayes for Ranking. In: Proceedings of the 5th IEEE International Conference on Data Mining, ICDM 2005, pp. 202-209. IEEE Computer Society Press, Los Alamitos (2005).

[25]

Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005), http://prdownloads.sourceforge.net/weka/datasets-UCI.jar

[26]

Merz, C., Murphy, P., Aha, D.: UCI repository of machine learning databases. In: Dept of ICS, University of California, Irvine (1997), http://www.ics.uci.edu/ mlearn/MLRepository.html

[27]

Nadeau, C., Bengio, Y.: Inference for the generalization error. In: Advances in Neural Information Processing Systems 12, pp. 307-313. MIT Press, Cambridge (1999).

[28]

Hand, D.J., Till, R.J.: A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning 45, 171-186 (2001).

[29]

Ling, C.X., Huang, J., Zhang, H.: AUC: a statistically consistent and more discriminating measure than accuracy. In: Proceedings of the International Joint Conference on Artificial Intelligence IJCAI03, Morgan Kaufmann, San Francisco (2003).

[30]

Lowd, D., Domingos, P.: Naive Bayes Models for Probability Estimation. In: Proceedings of the Twenty-Second International Conference on Machine Learning, pp. 529-536. ACM Press, New York (2005).

[31]

Jiang, L., Zhang, H.: Learning Naive Bayes for Probability Estimation by Feature Selection. In: Lamontagne, L., Marchand, M. (eds.) Canadian AI 2006. LNCS (LNAI), vol. 4013, pp. 503-514. Springer, Heidelberg (2006).

[32]

Grossman, D., Domingos, P.: Learning Bayesian Network Classifiers by Maximizing Conditional Likelihood. In: Proceedings of the Twenty-First International Conference on Machine Learning, pp. 361-368. ACM Press, Banff, Canada (2004).

[33]

Zhang, H., Su, J.: Naive Bayesian classifiers for ranking. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 501-512. Springer, Heidelberg (2004).

[34]

Zhang, H., Jiang, L., Su, J.: Augmenting Naive Bayes for Ranking. In: Proceedings of the 22nd International Conference on Machine Learning, ICML 2005, pp. 1025- 1032. ACM, New York (2005).

[35]

Jiang, L., Zhang, H., Cai, Z.: Discriminatively Improving Naive Bayes by Evolutionary Feature Selection. Romanian Journal of Information Science and Technology 9(3), 163-174 (2006).

Cited By

Zoppi TGazzini SCeccarelli A(2024)Anomaly-based error and intrusion detection in tabular dataFuture Generation Computer Systems10.1016/j.future.2024.06.051160:C(951-965)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1016/j.future.2024.06.051
Peretz OKoren MKoren O(2024)Naive Bayes classifier – An ensemble procedure for recall and precision enrichmentEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108972136:PBOnline publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.108972
Yang ZKeung JYu XXiao YJin ZZhang J(2023)On the Significance of Category Prediction for Code-Comment SynchronizationACM Transactions on Software Engineering and Methodology10.1145/353411732:2(1-41)Online publication date: 29-Mar-2023
https://dl.acm.org/doi/10.1145/3534117
Show More Cited By

Survey of Improving Naive Bayes for Classification
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches

Recommendations

A Novel Bayes Model: Hidden Naive Bayes

Because learning an optimal Bayesian network classifier is an NP-hard problem, learning-improved naive Bayes has attracted much attention from researchers. In this paper, we summarize the existing improved algorithms and propose a novel Bayes model: ...
Naive Bayes for optimal ranking

It is well known that naive Bayes performs surprisingly well in classification, but its probability estimation is poor. AUC (the area under the receiver operating characteristics curve) is a measure different from classification accuracy and probability ...
Averaged Naive Bayes Trees: A New Extension of AODE
ACML '09: Proceedings of the 1st Asian Conference on Machine Learning: Advances in Machine Learning

Naive Bayes (NB) is a simple Bayesian classifier that assumes the conditional independence and augmented NB (ANB) models are extensions of NB by relaxing the independence assumption. The averaged one-dependence estimators (AODE) is a classifier that ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

ADMA '07: Proceedings of the 3rd international conference on Advanced Data Mining and Applications

August 2007

632 pages

ISBN:9783540738701

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 06 August 2007

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zoppi TGazzini SCeccarelli A(2024)Anomaly-based error and intrusion detection in tabular dataFuture Generation Computer Systems10.1016/j.future.2024.06.051160:C(951-965)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1016/j.future.2024.06.051
Peretz OKoren MKoren O(2024)Naive Bayes classifier – An ensemble procedure for recall and precision enrichmentEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108972136:PBOnline publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.108972
Yang ZKeung JYu XXiao YJin ZZhang J(2023)On the Significance of Category Prediction for Code-Comment SynchronizationACM Transactions on Software Engineering and Methodology10.1145/353411732:2(1-41)Online publication date: 29-Mar-2023
https://dl.acm.org/doi/10.1145/3534117
Fu WZhang MJohnston M(2019)Bayesian genetic programming for edge detectionSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-018-3059-323:12(4097-4112)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.1007/s00500-018-3059-3
Balasaraswathi M Kalpana B (2018)Fast and Effective Classification using Parallel and Multi-start PSOJournal of Information Technology Research10.4018/JITR.201804010211:2(13-30)Online publication date: 1-Apr-2018
https://dl.acm.org/doi/10.4018/JITR.2018040102
Liu RZhang X(2018)Generating machine-executable plans from end-user's natural-language instructionsKnowledge-Based Systems10.1016/j.knosys.2017.10.023140:C(15-26)Online publication date: 15-Jan-2018
https://dl.acm.org/doi/10.1016/j.knosys.2017.10.023
Oliveira JCotacallapa MSeron WSantos RQuiles M(2016)Sentiment and Behavior Analysis of One Controversial American Individual on TwitterProceedings of the 23rd International Conference on Neural Information Processing - Volume 994810.1007/978-3-319-46672-9_57(509-518)Online publication date: 16-Oct-2016
https://dl.acm.org/doi/10.1007/978-3-319-46672-9_57
Tripathi NOakes MWermter S(2013)Hybrid classifiers based on semantic data subspaces for two-level text categorizationInternational Journal of Hybrid Intelligent Systems10.3233/HIS-13016310:1(33-41)Online publication date: 1-Jan-2013
https://dl.acm.org/doi/10.3233/HIS-130163
Ferreira RFreitas FBrito PMelo JLima RCosta E(2013)RetriBlogExpert Systems with Applications: An International Journal10.1016/j.eswa.2012.08.02040:4(1177-1195)Online publication date: 1-Mar-2013
https://dl.acm.org/doi/10.1016/j.eswa.2012.08.020
Ferreira RBrito PMelo JCosta ELima RFreitas FOssowski SLecca P(2012)An architecture-centered framework for developing blog crawlersProceedings of the 27th Annual ACM Symposium on Applied Computing10.1145/2245276.2231954(1131-1136)Online publication date: 26-Mar-2012
https://dl.acm.org/doi/10.1145/2245276.2231954
Show More Cited By

View Options

View options

Figures

Tables

Media

View Table of Conten