research-article

Adaptive Local Linear Discriminant Analysis

Authors:

Xuelong LiAuthors Info & Claims

ACM Transactions on Knowledge Discovery from Data (TKDD), Volume 14, Issue 1

Article No.: 9, Pages 1 - 19

https://doi.org/10.1145/3369870

Published: 03 February 2020 Publication History

Abstract

Dimensionality reduction plays a significant role in high-dimensional data processing, and Linear Discriminant Analysis (LDA) is a widely used supervised dimensionality reduction approach. However, a major drawback of LDA is that it is incapable of extracting the local structure information, which is crucial for handling multimodal data. In this article, we propose a novel supervised dimensionality reduction method named Adaptive Local Linear Discriminant Analysis (ALLDA), which adaptively learns a k-nearest neighbors graph from data themselves to extract the local connectivity of data. Furthermore, the original high-dimensional data usually contains noisy and redundant features, which has a negative impact on the evaluation of neighborships and degrades the subsequent classification performance. To address this issue, our method learns the similarity matrix and updates the subspace simultaneously so that the neighborships can be evaluated in the optimal subspaces where the noises have been removed. Through the optimal graph embedding, the underlying sub-manifolds of data in intra-class can be extracted precisely. Meanwhile, an efficient iterative optimization algorithm is proposed to solve the minimization problem. Promising experimental results on synthetic and real-world datasets are provided to evaluate the effectiveness of proposed method.

References

[1]

Mukund Balasubramanian and Eric L. Schwartz. 2002. The isomap algorithm and topological stability. Science 295, 5552 (2002), 7.

[2]

Deng Cai, Xiaofei He, Kun Zhou, Jiawei Han, and Hujun Bao. 2007. Locality sensitive discriminant analysis. In IJCAI, Vol. 2007. 1713--1726.

Digital Library

[3]

Hwann-Tzong Chen, Huang-Wei Chang, and Tyng-Luh Liu. 2005. Local discriminant embedding and its variants. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 2. IEEE, 846--853.

[4]

Zizhu Fan, Yong Xu, and David Zhang. 2011. Local linear discriminant analysis framework using sample neighbors. IEEE Transactions on Neural Networks 22, 7 (2011), 1119--1132.

Digital Library

[5]

Ronald A. Fisher. 1936. The use of multiple measurements in taxonomic problems. Annals of Human Genetics 7, 2 (1936), 179--188.

[6]

Keinosuke Fukunaga. 2013. Introduction to Statistical Pattern Recognition. Academic Press.

Digital Library

[7]

Quanxue Gao, Yunfang Huang, Hailin Zhang, Xin Hong, Kui Li, and Yong Wang. 2015. Discriminative sparsity preserving projections for image recognition. Pattern Recognition 48, 8 (2015), 2543--2553.

Digital Library

[8]

Athinodoros S. Georghiades, Peter N. Belhumeur, and David J. Kriegman. 2001. From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 6 (2001), 643--660.

Digital Library

[9]

Ping He, Xincheng Chang, Xiaohua Xu, Zhijun Zhang, Tianyu Jing, and Yuan Lou. 2019. Discriminative locally linear mapping for medical diagnosis. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-018-7064-4

[10]

Xiaofei He and Partha Niyogi. 2004. Locality preserving projections. In Advances in Neural Information Processing Systems. 153--160.

[11]

Thomas Hofmann and Joachim Buhmann. 1995. Multidimensional scaling and data clustering. In Advances in Neural Information Processing Systems. 459--466.

[12]

Di Hu, Feiping Nie, and Xuelong Li. 2018. Discrete spectral hashing for efficient similarity retrieval. IEEE Transactions on Image Processing 28, 3 (2018), 1080--1091.

[13]

Yi Huang, Dong Xu, and Feiping Nie. 2012a. Patch distribution compatible semisupervised dimension reduction for face and human gait recognition. IEEE Transactions on Circuits and Systems for Video Technology 22, 3 (2012), 479--488.

Digital Library

[14]

Yi Huang, Dong Xu, and Feiping Nie. 2012b. Semi-supervised dimension reduction using trace ratio criterion. IEEE Transactions on Neural Networks and Learning Systems 23, 3 (2012), 519--526.

[15]

Yangqing Jia, Feiping Nie, and Changshui Zhang. 2009. Trace ratio problem revisited. IEEE Transactions on Neural Networks 20, 4 (2009), 729--735.

Digital Library

[16]

Tae-Kyun Kim and Josef Kittler. 2005. Locally linear discriminant analysis for multimodally distributed classes for face recognition with a single model image. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 3 (2005), 318--327.

Digital Library

[17]

Bo Li, Zhang-Tao Fan, Xiao-Long Zhang, and De-Shuang Huang. 2019a. Robust dimensionality reduction via feature space to feature space distance metric learning. Neural Networks 112 (2019), 1--14. https://doi.org/10.1007/s00521-019-04055-6

[18]

Haifeng Li, Tao Jiang, and Keshu Zhang. 2004. Efficient and robust feature extraction by maximum margin criterion. In Advances in Neural Information Processing Systems. 97--104.

[19]

Xuelong Li, Mulin Chen, Feiping Nie, and Qi Wang. 2017. Locality adaptive discriminant analysis. In 26th International Joint Conference on Artificial Intelligence. AAAI Press, 2201--2207.

[20]

Xiaoyan Li, Lefei Zhang, and Jane You. 2019c. Locally weighted discriminant analysis for hyperspectral image classification. Remote Sensing 11, 2 (2019), 109.

[21]

Yunsong Li, Chiru Ge, Weiwei Sun, Jiangtao Peng, Qian Du, and Keyan Wang. 2019b. Hyperspectral and LiDAR data fusion classification using superpixel segmentation-based local pixel neighborhood preserving embedding. Remote Sensing 11, 5 (2019), 550.

[22]

Zimo Liu, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. 2019. Person reidentification by joint local distance metric and feature transformation. IEEE Transactions on Neural Networks and Learning Systems 30, 10 (2019), 2999--3009.

[23]

Minnan Luo, Lingling Zhang, Feiping Nie, Xiaojun Chang, Buyue Qian, and Qinghua Zheng. 2017. Adaptive semi-supervised learning with discriminative least squares regression. In 26th International Joint Conference on Artificial Intelligence. AAAI Press, 2421--2427.

[24]

Tingjin Luo, Chenping Hou, Feiping Nie, and Dongyun Yi. 2018. Dimension reduction for non-Gaussian data by adaptive discriminative analysis. IEEE Transactions on Cybernetics 49, 3 (2018), 933--946.

[25]

Sameer A. Nene, Shree K. Nayar, and Hiroshi Murase. 1996. Columbia object image library (coil-20). Technical Report CUCS-005-96.

[26]

Feiping Nie, Zheng Wang, Rong Wang, and Xuelong Li. 2019. Submanifold-preserving discriminant analysis with an auto-optimized graph. IEEE Transactions on Cybernetics (2019), 1--14. https://doi.org/10.1109/TCYB.2019.2910751

[27]

Feiping Nie, Shiming Xiang, Yangqiu Song, and Changshui Zhang. 2009. Extracting the optimal dimensionality for local tensor discriminant analysis. Pattern Recognition 42, 1 (2009), 105--114.

Digital Library

[28]

Feiping Nie, Shiming Xiang, and Changshui Zhang. 2007. Neighborhood MinMax projections. In IJCAI. 993--998.

[29]

Aude Oliva and Antonio Torralba. 2001. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision 42, 3 (2001), 145--175.

Digital Library

[30]

D. Parikh and K. Grauman. 2011. Relative attributes. In IEEE International Conference on Computer Vision (ICCV’11).

[31]

Lishan Qiao, Songcan Chen, and Xiaoyang Tan. 2010. Sparsity preserving projections with applications to face recognition. Pattern Recognition 43, 1 (2010), 331--341.

Digital Library

[32]

C. Radhakrishna Rao. 1948. The utilization of multiple measurements in problems of biological classification. Journal of the Royal Statistical Society. Series B (Methodological) 10, 2 (1948), 159--203.

[33]

Sam T. Roweis and Lawrence K. Saul. 2000. Nonlinear dimensionality reduction by locally linear embedding. Science 290, 5500 (2000), 2323--2326.

[34]

Ronghua Shang, Yang Meng, Wenbing Wang, Fanhua Shang, and Licheng Jiao. 2019. Local discriminative based sparse subspace learning for feature selection. Pattern Recognition 92 (2019), 219--230. https://doi.org/10.1109/TCYB.2019.2910751

Digital Library

[35]

Masashi Sugiyama. 2006. Local fisher discriminant analysis for supervised dimensionality reduction. In 23rd International Conference on Machine Learning. ACM, 905--912.

Digital Library

[36]

Masashi Sugiyama. 2007. Dimensionality reduction of multimodal labeled data by local Fisher discriminant analysis. Journal of Machine Learning Research 8, May (2007), 1027--1061.

[37]

Yao Tang, Ruilin Li, Yuhang Liu, and Jufu Feng. 2019. FClassNet: A fingerprint classification network integrated with the domain knowledge. Science China Information Sciences 62, 12 (2019), 229102.

[38]

Zhenyu Tang, Yue Cui, and Bo Jiang. 2017. Groupwise registration of MR brain images containing tumors via spatially constrained low-rank based image recovery. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 397--405.

[39]

Matthew A. Turk and Alex P. Pentland. 1991. Face recognition using eigenfaces. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’91). IEEE, 586--591.

[40]

Guoqiang Wang and Nianfeng Shi. 2019. Collaborative representation-based discriminant neighborhood projections for face recognition. Neural Computing and Applications. https://doi.org/10.1007/s00521-019-04055-6

[41]

Rong Wang, Feiping Nie, Richang Hong, Xiaojun Chang, Xiaojun Yang, and Weizhong Yu. 2017. Fast and orthogonal locality preserving projections for dimensionality reduction. IEEE Transactions on Image Processing 26, 10 (2017), 5019--5030.

Digital Library

[42]

Svante Wold, Kim Esbensen, and Paul Geladi. 1987. Principal component analysis. Chemometrics and Intelligent Laboratory Systems 2, 1--3 (1987), 37--52.

[43]

Shiming Xiang, Feiping Nie, Changshui Zhang, and Chunxia Zhang. 2009. Nonlinear dimensionality reduction with local spline embedding. IEEE Transactions on Knowledge and Data Engineering 21, 9 (2009), 1285--1298.

Digital Library

[44]

Guowen Xu, Hongwei Li, Yuanshun Dai, Kan Yang, and Xiaodong Lin. 2019. Enabling efficient and geometric range query with access control over encrypted spatial data. IEEE Transactions on Information Forensics and Security 14, 4 (2018), 870--885.

[45]

Jie Xu, Lei Luo, Cheng Deng, and Heng Huang. 2018a. Bilevel distance metric learning for robust image recognition. In Advances in Neural Information Processing Systems. 4198--4207.

[46]

Jie Xu, Lei Luo, Cheng Deng, and Heng Huang. 2018b. New robust metric learning model using maximum correntropy criterion. In 24th ACM SIGKDD International Conference on Knowledge Discovery 8 Data Mining. ACM, 2555--2564.

Digital Library

[47]

Shuicheng Yan, Dong Xu, Benyu Zhang, and Hong-Jiang Zhang. 2005. Graph embedding: A general framework for dimensionality reduction. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 2. IEEE, 830--837.

[48]

Wankou Yang, Changyin Sun, and Lei Zhang. 2011. A multi-manifold discriminant analysis method for image feature extraction. Pattern Recognition 44, 8 (2011), 1649--1657.

Digital Library

[49]

Yi Yang, Feiping Nie, Dong Xu, Jiebo Luo, Yueting Zhuang, and Yunhe Pan. 2012. A multimedia retrieval framework based on semi-supervised ranking and relevance feedback. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 4 (2012), 723--742.

Digital Library

[50]

Qiaolin Ye, Liyong Fu, Zhao Zhang, Henghao Zhao, and Meem Naiem. 2018. Lp-and Ls-norm distance based robust linear discriminant analysis. Neural Networks 105 (2018), 393--404. https://doi.org/10.1016/j.neunet.2018.05.020

[51]

Lin Zhang, Lei Zhang, David Zhang, and Zhenhua Guo. 2012. Phase congruency induced local features for finger-knuckle-print recognition. Pattern Recognition 45, 7 (2012), 2522--2531.

Digital Library

[52]

Haifeng Zhao, Zheng Wang, and Feiping Nie. 2016. Orthogonal least squares regression for feature extraction. Neurocomputing 216, C (2016), 200--207.

[53]

Haifeng Zhao, Zheng Wang, and Feiping Nie. 2018a. Adaptive neighborhood MinMax projections. Neurocomputing 313 (2018), 155--166. https://doi.org/10.1016/j.neucom.2018.06.045

[54]

Haifeng Zhao, Zheng Wang, and Feiping Nie. 2018b. A new formulation of linear discriminant analysis for robust dimensionality reduction. IEEE Transactions on Knowledge and Data Engineering 31, 4 (2018), 629--640.

Digital Library

Cited By

Yang XCao CZhou KPeng SWang ZLin LNie F(2025)A novel linear discriminant analysis based on alternate ratio sum minimizationInformation Sciences10.1016/j.ins.2024.121444689(121444)Online publication date: Jan-2025
https://doi.org/10.1016/j.ins.2024.121444
Zhang TChen YSun ZHuang LDai QShen Q(2024)Fault diagnosis of rolling bearing based on hierarchical discrete entropy and semi-supervised local Fisher discriminant analysisJournal of Vibroengineering10.21595/jve.2024.2394526:6(1317-1335)Online publication date: 22-Aug-2024
https://doi.org/10.21595/jve.2024.23945
Li XWang H(2024)On Mean-Optimal Robust Linear Discriminant AnalysisACM Transactions on Knowledge Discovery from Data10.1145/366550018:8(1-27)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3665500
Show More Cited By

Index Terms

Adaptive Local Linear Discriminant Analysis
1. Computing methodologies
  1. Machine learning
    1. Machine learning algorithms
      1. Feature selection

Recommendations

Graph regularized linear discriminant analysis and its generalization

Linear discriminant analysis (LDA) is a powerful dimensionality reduction technique, which has been widely used in many applications. Although, LDA is well-known for its discriminant capability, it clearly does not capture the geometric structure of the ...
Linear discriminant analysis with worst between-class separation and average within-class compactness

Linear discriminant analysis (LDA) is one of the most popular supervised dimensionality reduction (DR) techniques and obtains discriminant projections by maximizing the ratio of average-case between-class scatter to average-case within-class scatter. ...
A Two-Stage Linear Discriminant Analysis via QR-Decomposition

Linear Discriminant Analysis (LDA) is a well-known method for feature extraction and dimension reduction. It has been used widely in many applications involving high-dimensional data, such as image and text classification. An intrinsic limitation of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Knowledge Discovery from Data

ACM Transactions on Knowledge Discovery from Data Volume 14, Issue 1

February 2020

325 pages

ISSN:1556-4681

EISSN:1556-472X

DOI:10.1145/3375789

Editors:
Charu Aggarwal
IBM T. J. Watson Research, USA
,
Xindong Wu
Minginglamp Academy of Sciences, China

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 February 2020

Accepted: 01 October 2019

Revised: 01 June 2019

Received: 01 September 2018

Published in TKDD Volume 14, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Key Research and Development Program of China under Grant
Fundamental Research Funds for the Central Universities
National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

46
Total Citations
View Citations
920
Total Downloads

Downloads (Last 12 months)103
Downloads (Last 6 weeks)13

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang XCao CZhou KPeng SWang ZLin LNie F(2025)A novel linear discriminant analysis based on alternate ratio sum minimizationInformation Sciences10.1016/j.ins.2024.121444689(121444)Online publication date: Jan-2025
https://doi.org/10.1016/j.ins.2024.121444
Zhang TChen YSun ZHuang LDai QShen Q(2024)Fault diagnosis of rolling bearing based on hierarchical discrete entropy and semi-supervised local Fisher discriminant analysisJournal of Vibroengineering10.21595/jve.2024.2394526:6(1317-1335)Online publication date: 22-Aug-2024
https://doi.org/10.21595/jve.2024.23945
Li XWang H(2024)On Mean-Optimal Robust Linear Discriminant AnalysisACM Transactions on Knowledge Discovery from Data10.1145/366550018:8(1-27)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3665500
Wang ZNie FZhang CWang RLi X(2024)Worst-Case Discriminative Feature Learning via Max-Min Ratio AnalysisIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.332345346:1(641-658)Online publication date: Jan-2024
https://doi.org/10.1109/TPAMI.2023.3323453
Wang ZNie FWang HHuang HWang F(2024)Toward Robust Discriminative Projections Learning Against Adversarial Patch AttacksIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.332160635:12(18784-18798)Online publication date: Dec-2024
https://doi.org/10.1109/TNNLS.2023.3321606
Nie HLi QWang ZZhao HNie F(2024)Semisupervised Subspace Learning With Adaptive Pairwise Graph EmbeddingIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.331178935:12(18105-18119)Online publication date: Dec-2024
https://doi.org/10.1109/TNNLS.2023.3311789
Nie FChen HXiang SZhang CYan SLi X(2024)On the Equivalence of Linear Discriminant Analysis and Least Squares RegressionIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.3208944(1-11)Online publication date: 2024
https://doi.org/10.1109/TNNLS.2022.3208944
Wang ZLi QNie FWang RWang FLi X(2024)Efficient Local Coherent Structure Learning via Self-Evolution Bipartite GraphIEEE Transactions on Cybernetics10.1109/TCYB.2023.332184354:8(4527-4538)Online publication date: Aug-2024
https://doi.org/10.1109/TCYB.2023.3321843
Wang JLiang YMa J(2024)Innovative strategies for intelligent services in smart libraries in the information age based on linear discriminant analysisSystems and Soft Computing10.1016/j.sasc.2024.2001596(200159)Online publication date: Dec-2024
https://doi.org/10.1016/j.sasc.2024.200159
Wang JYin HNie FLi X(2024)Adaptive and fuzzy locality discriminant analysis for dimensionality reductionPattern Recognition10.1016/j.patcog.2024.110382151(110382)Online publication date: Jul-2024
https://doi.org/10.1016/j.patcog.2024.110382
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents