article

Statistical independence and novelty detection with information preserving nonlinear maps

Authors:

Stefan MiesbachAuthors Info & Claims

Neural Computation, Volume 8, Issue 2

Pages 260 - 269

https://doi.org/10.1162/neco.1996.8.2.260

Published: 01 February 1996 Publication History

Abstract

According to Barlow (1989), feature extraction can be understood as finding a statistically independent representation of the probability distribution underlying the measured signals. The search for a statistically independent representation can be formulated by the criterion of minimal mutual information, which reduces to decorrelation in the case of gaussian distributions. If nongaussian distributions are to be considered, minimal mutual information is the appropriate generalization of decorrelation as used in linear Principal Component Analyses (PCA). We also generalize to nonlinear transformations by only demanding perfect transmission of information. This leads to a general class of nonlinear transformations, namely symplectic maps. Conservation of information allows us to consider only the statistics of single coordinates. The resulting factorial representation of the joint probability distribution gives a density estimation. We apply this concept to the real world problem of electrical motor fault detection treated as a novelty detection task.

References

[1]

Abraham, R., and Marsden, J. 1978. Foundations of Mechanics. Benjamin-Cummings, London.

[2]

Barlow, H. 1989. Unsupervised learning. Neural Comp. 1(1), 295-311.

Digital Library

[3]

Bell, A. J., and Sejnowski, T. 1995. An information-maximization approach to blind separation and blind deconvolution. Neural Comp. 7(6), 1129-1159.

Digital Library

[4]

Comon, P. 1994. Independent component analysis, a new concept. Signal Processing 36, 287-314.

Digital Library

[5]

Deco, G., and Brauer, W. 1995. Nonlinear higher order statistical decorrelation by volume conserving neural architectures. Neural Networks (in press).

[6]

Deco, G., and Parra, L. 1996. Nonlinear features extraction by redundancy reduction with stochastic neural networks. Neural Networks (to appear).

[7]

Deco, G., and Schürman, B. 1995. Learning time series evolution by unsupervised extraction of correlations. Phys. Rev. E 51(2), 1780-1790.

[8]

Duda, R. O., and Hart, P. E. 1973. Pattern Classification and Scene Analysis. Wiley, New York.

[9]

Feng Kang, and Qin Meng-zhao. 1985. The symplectic methods for the computation of Hamiltonian equations. In Numerical Methods for Partial Differential Equations, Proceedings of a Conference held in Shanghai, 1987. Lecture Notes in Mathematics. Zhu You-lan and Guo Ben-yu, eds., Vol. 1297, pp. 1- 35. Springer, Berlin.

[10]

Hornik, K., Stinchcombe, M., and White, H. 1989. Multilayer feedforward neural networks are universal approximators. Neural Networks 2, 359-366.

Digital Library

[11]

Miesbach, S., and Pesch, H. J. 1992. Symplectic phase flow approximation for the numerical integration of canonical systems. Numer. Math. 61, 501-521.

Digital Library

[12]

Nadal, J-P., and Parga, N. 1994. Non-linear neurons in the low noise limit: A factorial code maximizes information transfer. Network 5(4), 565-581.

[13]

Papoulis, A. 1991. Probability, Random Variables, and Stochastic Processes, 3rd ed. McGraw-Hill, New York.

[14]

Parra, L., Deco, G., and Miesbach, S. 1995. Redundancy reduction with information preserving nonlinear maps. Network 6, 61-72.

[15]

Redlich, A. N. 1993. Supervised factorial learning. Neural Comp. 5, 750-766.

Digital Library

[16]

Schmidhuber, J. 1992. Learning factorial codes by predictability minimization. Neural Comp. 4(6), 863-879.

Digital Library

[17]

Shannon, C. 1948. A mathematical theory of communication. Bell Syst. Tech. J. 7, 379-423.

[18]

Siegel 1943. Symplectic geometry. Am. J. Math. 65, 1-86.

[19]

Stoer, J., and Bulirsch, R. 1993. Introduction to Numerical Analysis. Springer, New York.

Cited By

Nheri SKsantini RKaâniche MBouhoula A(2023)Exploiting scatter matrix on one-class support vector machine based on low variance directionIntelligent Data Analysis10.3233/IDA-22703627:6(1663-1679)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.3233/IDA-227036
Pang GCao L(2020)Heterogeneous Univariate Outlier Ensembles in Multidimensional DataACM Transactions on Knowledge Discovery from Data10.1145/340393414:6(1-27)Online publication date: 28-Sep-2020
https://dl.acm.org/doi/10.1145/3403934
Liu JMiao QSun YSong JQuan Y(2016)Fast structural ensemble for One-Class ClassificationPattern Recognition Letters10.1016/j.patrec.2016.06.02880:C(179-187)Online publication date: 1-Sep-2016
https://dl.acm.org/doi/10.1016/j.patrec.2016.06.028
Show More Cited By

Recommendations

Almost k-wise independence versus k-wise independence

We say that a distribution over {0,1}ⁿ is (ε,k)-wise independent if its restriction to every k coordinates results in a distribution that is ε-close to the uniform distribution. A natural question regarding (ε, k)-wise independent distributions is how ...
Novelty Detection with Multivariate Extreme Value Statistics

Novelty detection, or one-class classification, aims to determine if data are "normal" with respect to some model of normality constructed using examples of normal system behaviour. If that model is composed of generative probability distributions, the ...
A kernel statistical test of independence
NIPS'07: Proceedings of the 20th International Conference on Neural Information Processing Systems

Although kernel measures of independence have been widely applied in machine learning (notably in kernel ICA), there is as yet no method to determine whether they have detected statistically significant dependence. We provide a novel test of the ...

Comments

Information & Contributors

Information

Published In

cover image Neural Computation

Neural Computation Volume 8, Issue 2

February 15, 1996

246 pages

ISSN:0899-7667

Issue’s Table of Contents

Publisher

MIT Press

Cambridge, MA, United States

Publication History

Published: 01 February 1996

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Nheri SKsantini RKaâniche MBouhoula A(2023)Exploiting scatter matrix on one-class support vector machine based on low variance directionIntelligent Data Analysis10.3233/IDA-22703627:6(1663-1679)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.3233/IDA-227036
Pang GCao L(2020)Heterogeneous Univariate Outlier Ensembles in Multidimensional DataACM Transactions on Knowledge Discovery from Data10.1145/340393414:6(1-27)Online publication date: 28-Sep-2020
https://dl.acm.org/doi/10.1145/3403934
Liu JMiao QSun YSong JQuan Y(2016)Fast structural ensemble for One-Class ClassificationPattern Recognition Letters10.1016/j.patrec.2016.06.02880:C(179-187)Online publication date: 1-Sep-2016
https://dl.acm.org/doi/10.1016/j.patrec.2016.06.028
Saki FKehtarnavaz N(2016)Online frame-based clustering with unknown number of clustersPattern Recognition10.1016/j.patcog.2016.03.01057:C(70-83)Online publication date: 1-Sep-2016
https://dl.acm.org/doi/10.1016/j.patcog.2016.03.010
Liu JMiao QSun YSong JQuan Y(2016)Modular ensembles for one-class classification based on density analysisNeurocomputing10.1016/j.neucom.2015.06.037171:C(262-276)Online publication date: 1-Jan-2016
https://dl.acm.org/doi/10.1016/j.neucom.2015.06.037
(2015)Gear classification and fault detection using a diffusion map frameworkPattern Recognition Letters10.1016/j.patrec.2014.10.01953:C(53-61)Online publication date: 1-Feb-2015
https://dl.acm.org/doi/10.1016/j.patrec.2014.10.019
Wang ZZhao ZWeng SZhang C(2015)Solving one-class problem with outlier examples by SVMNeurocomputing10.1016/j.neucom.2014.03.072149:PA(100-105)Online publication date: 3-Feb-2015
https://dl.acm.org/doi/10.1016/j.neucom.2014.03.072
Mefraz Khan NKsantini RShafiq Ahmad IGuan L(2014)Covariance-guided One-Class Support Vector MachinePattern Recognition10.1016/j.patcog.2014.01.00447:6(2165-2177)Online publication date: 1-Jun-2014
https://dl.acm.org/doi/10.1016/j.patcog.2014.01.004
Abin ABeigy H(2014)Active selection of clustering constraintsPattern Recognition10.1016/j.patcog.2013.09.03447:3(1443-1458)Online publication date: 1-Mar-2014
https://dl.acm.org/doi/10.1016/j.patcog.2013.09.034
Le TTran DMa WSharma D(2011)Multiple distribution data description learning algorithm for novelty detectionProceedings of the 15th Pacific-Asia conference on Advances in knowledge discovery and data mining - Volume Part II10.5555/2022850.2022871(246-257)Online publication date: 24-May-2011
https://dl.acm.org/doi/10.5555/2022850.2022871
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents