research-article

Simple and deterministic matrix sketching

Author:

Edo LibertyAuthors Info & Claims

KDD '13: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 581 - 588

https://doi.org/10.1145/2487575.2487623

Published: 11 August 2013 Publication History

Abstract

A sketch of a matrix A is another matrix B which is significantly smaller than A but still approximates it well. Finding such sketches efficiently is an important building block in modern algorithms for approximating, for example, the PCA of massive matrices. This task is made more challenging in the streaming model, where each row of the input matrix can only be processed once and storage is severely limited.

In this paper we adapt a well known streaming algorithm for approximating item frequencies to the matrix sketching setting. The algorithm receives n rows of a large matrix A ε ℜ ^{n x m} one after the other in a streaming fashion. It maintains a sketch B ℜ ^{l x m} containing only l << n rows but still guarantees that A^TA B^TB. More accurately, ∀x || x,||=1 0≤||Ax||² - ||Bx||² ≤ 2||A||_f ² l Or B^TB prec A^TA and ||A^TA - B^TB|| ≤ 2 ||A||f² l.

This gives a streaming algorithm whose error decays proportional to 1/l using O(ml) space. For comparison, random-projection, hashing or sampling based algorithms produce convergence bounds proportional to 1/√l. Sketch updates per row in A require amortized O(ml) operations and the algorithm is perfectly parallelizable. Our experiments corroborate the algorithm's scalability and improved convergence rate. The presented algorithm also stands out in that it is deterministic, simple to implement and elementary to prove.

References

[1]

Dimitris Achlioptas. Database-friendly random projections. In Proceedings of the twentieth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, PODS '01, pages 274--281, New York, NY, USA, 2001. ACM.

Digital Library

[2]

Dimitris Achlioptas and Frank Mcsherry. Fast computation of low-rank matrix approximations. J. ACM, 54(2), 2007.

Digital Library

[3]

Rudolf Ahlswede and Andreas Winter. Strong converse for identification via quantum channels. IEEE Transactions on Information Theory, 48(3):569--579, 2002.

Digital Library

[4]

Sanjeev Arora, Elad Hazan, and Satyen Kale. A fast random sampling algorithm for sparsifying matrices. In Proceedings of the 9th international conference on Approximation Algorithms for Combinatorial Optimization Problems, and 10th international conference on Randomization and Computation, APPROX'06/RANDOM'06, pages 272--279, Berlin, Heidelberg, 2006. Springer-Verlag.

Digital Library

[5]

Christos Boutsidis, Petros Drineas, and Malik Magdon-Ismail. Near optimal column-based matrix reconstruction. In Proceedings of the 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science, FOCS '11, pages 305--314, Washington, DC, USA, 2011. IEEE Computer Society.

Digital Library

[6]

Christos Boutsidis, Michael W. Mahoney, and Petros Drineas. An improved approximation algorithm for the column subset selection problem. In Proceedings of the twentieth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '09, pages 968--977, Philadelphia, PA, USA, 2009. Society for Industrial and Applied Mathematics.

Digital Library

[7]

Kenneth L. Clarkson and David P. Woodruff. Numerical linear algebra in the streaming model. In Proceedings of the 41st annual ACM symposium on Theory of computing, STOC '09, pages 205--214, New York, NY, USA, 2009. ACM.

Digital Library

[8]

Kenneth L. Clarkson and David P. Woodruff. Low rank approximation and regression in input sparsity time. In Proceedings of the 45th annual ACM symposium on Symposium on theory of computing, STOC '13, pages 81--90, New York, NY, USA, 2013. ACM.

Digital Library

[9]

Anirban Dasgupta, Ravi Kumar, and Tamás Sarlós. A sparse johnson: Lindenstrauss transform. In STOC, pages 341--350, 2010.

Digital Library

[10]

Erik D. Demaine, Alejandro López-Ortiz, and J. Ian Munro. Frequency estimation of internet packet streams with limited space. In Proceedings of the 10th Annual European Symposium on Algorithms, ESA '02, pages 348--360, London, UK, UK, 2002. Springer-Verlag.

Digital Library

[11]

Amit Deshpande and Santosh Vempala. Adaptive sampling and fast low-rank matrix approximation. In APPROX-RANDOM, pages 292--303, 2006.

Digital Library

[12]

Petros Drineas and Ravi Kannan. Pass efficient algorithms for approximating large matrices. In Proceedings of the fourteenth annual ACM-SIAM symposium on Discrete algorithms, SODA '03, pages 223--232, Philadelphia, PA, USA, 2003. Society for Industrial and Applied Mathematics.

Digital Library

[13]

Petros Drineas, Michael W. Mahoney, and S. Muthukrishnan. Relative-error cur matrix decompositions. SIAM J. Matrix Analysis Applications, 30(2):844--881, 2008.

Digital Library

[14]

Petros Drineas, Michael W. Mahoney, S. Muthukrishnan, and Tamas Sarlos. Faster least squares approximation. Numer. Math., 117(2):219--249, February 2011.

Digital Library

[15]

Petros Drineas and Anastasios Zouzias. A note on element-wise matrix sparsification via a matrix-valued bernstein inequality. Inf. Process. Lett., 111(8):385--389, March 2011.

Digital Library

[16]

Alan Frieze, Ravi Kannan, and Santosh Vempala. Fast monte-carlo algorithms for finding low-rank approximations. In Proceedings of the 39th Annual Symposium on Foundations of Computer Science, FOCS '98, pages 370--, Washington, DC, USA, 1998. IEEE Computer Society.

Digital Library

[17]

Phillip B. Gibbons and Yossi Matias. External memory algorithms, 1999.

[18]

Daniel M. Kane and Jelani Nelson. Sparser johnson-lindenstrauss transforms. In SODA, pages 1195--1206, 2012.

Digital Library

[19]

Richard M. Karp, Scott Shenker, and Christos H. Papadimitriou. A simple algorithm for finding frequent elements in streams and bags. ACM Trans. Database Syst., 28(1):51--55, March 2003.

Digital Library

[20]

Edo Liberty. www.cs.yale.edu/homes/el327/public/experimentalresults/.

[21]

Edo Liberty, Franco Woolfe, Per-Gunnar Martinsson, Vladimir Rokhlin, and Mark Tygert. Randomized algorithms for the low-rank approximation of matrices. Proceedings of the National Academy of Sciences, 104(51):20167--20172, December 2007.

[22]

Michael W. Mahoney, Petros Drineas, Malik Magdon-Ismail, and David P. Woodruff. Fast approximation of matrix coherence and statistical leverage. In ICML, 2012.

[23]

Jayadev Misra and David Gries. Finding repeated elements. Technical report, Ithaca, NY, USA, 1982.

Digital Library

[24]

Roberto Imbuzeiro Oliveira. Sums of random hermitian matrices and an inequality by rudelson. arXiv:1004.3821v1, April 2010.

[25]

Christos H. Papadimitriou, Hisao Tamaki, Prabhakar Raghavan, and Santosh Vempala. Latent semantic indexing: a probabilistic analysis. In Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systems, PODS '98, pages 159--168, New York, NY, USA, 1998. ACM.

Digital Library

[26]

Mark Rudelson and Roman Vershynin. Sampling from large matrices: An approach through geometric functional analysis. J. ACM, 54(4), July 2007.

Digital Library

[27]

Tamas Sarlos. Improved approximation algorithms for large matrices via random projections. In FOCS, pages 143--152, 2006.

Digital Library

[28]

S. S. Vempala. The Random Projection Method. American Mathematical Society, 2004.

[29]

Roman Vershynin. A note on sums of independent random matrices after ahlswede-winter. Lecture Notes.

[30]

Roman Vershynin. Spectral norm of products of random and deterministic matrices. Probability Theory and Related Fields, 150(3--4):471--509, 2011.

[31]

Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pages 1113--1120, New York, NY, USA, 2009. ACM.

Digital Library

Cited By

Chen DNie MWang ZChen HWang D(2024)A Negative Sample-Free Graph Contrastive Learning AlgorithmMathematics10.3390/math1210158112:10(1581)Online publication date: 18-May-2024
https://doi.org/10.3390/math12101581
Yin HWen DLi JWei ZZhang XHuang ZLi F(2024)Optimal Matrix Sketching over Sliding WindowsProceedings of the VLDB Endowment10.14778/3665844.366584717:9(2149-2161)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.14778/3665844.3665847
Yao ZLi LChen MWei XChen CBaeza-Yates RBonchi F(2024)Approximate Matrix Multiplication over Sliding WindowsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671819(3896-3906)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671819
Show More Cited By

Index Terms

Simple and deterministic matrix sketching
1. Computing methodologies
  1. Symbolic and algebraic manipulation
    1. Symbolic and algebraic algorithms
      1. Linear algebra algorithms
2. Mathematics of computing
  1. Mathematical analysis
    1. Numerical analysis
      1. Computations on matrices

Recommendations

Matrix Sketching Over Sliding Windows
SIGMOD '16: Proceedings of the 2016 International Conference on Management of Data

Large-scale matrix computation becomes essential for many data data applications, and hence the problem of sketching matrix with small space and high precision has received extensive study for the past few years. This problem is often considered in the ...
Separations and equivalences between turnstile streaming and linear sketching
STOC 2020: Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing

A longstanding observation, which was partially proven by Li, Nguyen, and Woodruff in 2014, and extended by Ai, Hu, Li, and Woodruff in 2016, is that any turnstile streaming algorithm can be implemented as a linear sketch (the reverse is trivially true)...
On Sketching Quadratic Forms
ITCS '16: Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science
We undertake a systematic study of sketching a quadratic form: given an n x n matrix A, create a succinct sketch sk(A) which can produce (without further access to A) a multiplicative (1+ε)-approximation to x^T A x for any desired query x ∈ Rⁿ. While a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '13: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining

August 2013

1534 pages

ISBN:9781450321747

DOI:10.1145/2487575

Editors:
Rayid Ghani
University of Chicago
,
Ted E. Senator
SAIC
,
Paul Bradley
MethodCare, Inc.
,
Rajesh Parekh
Groupon
,
Jingrui He
Stevens Institute of Technology
,
General Chairs:
Robert L. Grossman
University of Chicago and Open Data Group
,
Ramasamy Uthurusamy
General Motors Corporation (retired)
,
Program Chairs:
Inderjit S. Dhillon
University of Texas
,
Yehuda Koren
Google

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 August 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD' 13

Sponsor:

KDD' 13: The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 11 - 14, 2013

Illinois, Chicago, USA

Acceptance Rates

KDD '13 Paper Acceptance Rate 125 of 726 submissions, 17%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

155
Total Citations
View Citations
1,770
Total Downloads

Downloads (Last 12 months)81
Downloads (Last 6 weeks)6

Reflects downloads up to 10 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen DNie MWang ZChen HWang D(2024)A Negative Sample-Free Graph Contrastive Learning AlgorithmMathematics10.3390/math1210158112:10(1581)Online publication date: 18-May-2024
https://doi.org/10.3390/math12101581
Yin HWen DLi JWei ZZhang XHuang ZLi F(2024)Optimal Matrix Sketching over Sliding WindowsProceedings of the VLDB Endowment10.14778/3665844.366584717:9(2149-2161)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.14778/3665844.3665847
Yao ZLi LChen MWei XChen CBaeza-Yates RBonchi F(2024)Approximate Matrix Multiplication over Sliding WindowsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671819(3896-3906)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671819
Yi QWang CWang KWang Y(2024)Effective Streaming Low-Tubal-Rank Tensor Approximation via Frequent DirectionsIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.318109735:1(1113-1126)Online publication date: Jan-2024
https://doi.org/10.1109/TNNLS.2022.3181097
Wang SFeng KLi CYuan YWang G(2024)Learning to Generate Parameters of ConvNets for Unseen Image DataIEEE Transactions on Image Processing10.1109/TIP.2024.344573133(5577-5592)Online publication date: 2024
https://doi.org/10.1109/TIP.2024.3445731
Wu AYuan YLi CMa YZhang H(2024)Attributed Network Embedding in Streaming Style2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00243(3138-3150)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00243
Reyhani Hamedani MRyu JKim S(2024)Random-Walk-Based or Similarity-Based Methods, Which is Better for Directed Graph Embedding?2024 IEEE International Conference on Big Data and Smart Computing (BigComp)10.1109/BigComp60711.2024.00022(83-89)Online publication date: 18-Feb-2024
https://doi.org/10.1109/BigComp60711.2024.00022
Yang XTian PCheng XWan YSong M(2024)Regularized online exponentially concave optimizationNeurocomputing10.1016/j.neucom.2024.127789595(127789)Online publication date: Aug-2024
https://doi.org/10.1016/j.neucom.2024.127789
Dufrenois FKhatib AHamlich MHamad D(2024)Collaborative and dynamic kernel discriminant analysis for large-scale problems: applications in multi-class learning and novelty detectionProgress in Artificial Intelligence10.1007/s13748-023-00309-6Online publication date: 22-Jan-2024
https://doi.org/10.1007/s13748-023-00309-6
Zhou S(2024)Streaming First‐Order Optimization Methods With Momentum for Dominant Eigenspace EstimationNumerical Linear Algebra with Applications10.1002/nla.2595Online publication date: 16-Oct-2024
https://doi.org/10.1002/nla.2595
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents