Article

Generalized component analysis for text with heterogeneous attributes

Authors:

Andrew McCallumAuthors Info & Claims

KDD '07: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 794 - 803

https://doi.org/10.1145/1281192.1281277

Published: 12 August 2007 Publication History

Abstract

We present a class of richly structured, undirected hidden variable models suitable for simultaneously modeling text along with other attributes encoded in different modalities. Our model generalizes techniques such as principal component analysis to heterogeneous data types. In contrast to other approaches, this framework allows modalities such as words, authors and timestamps to be captured in their natural, probabilistic encodings. A latent space representation for a previously unseen document can be obtained through a fast matrix multiplication using our method. We demonstrate the effectiveness of our framework on the task of author prediction from 13 years of the NIPS conference proceedings and for a recipient prediction task using a 10-month academic email archive of a researcher. Our approach should be more broadly applicable to many real-world applications where one wishes to efficiently make predictions for a large number of potential outputs using dimensionality reduction in a well defined probabilistic framework.

Supplementary Material

JPG File (p794-wang-200.jpg)

Download
11.07 KB

JPG File (p794-wang-768.jpg)

Download
16.14 KB

Low Resolution (p794-wang-200.mov)

Download
31.32 MB

High Resolution (p794-wang-768.mov)

Download
109.04 MB

References

[1]

C. Andrieu, N. de Freitas, A. Doucet, and M. Jordan. An introduction to MCMC for machine learning. Machine Learning, 50:5--43, 2003.

[2]

D. Blei and J. Lafferty. Dynamic topic models. In Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA, 2006.

Digital Library

[3]

D. Blei, A. Ng, and M. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993--1022, 2003.

Digital Library

[4]

W. Buntine and A. Jakulin. Applying discrete PCA in data analysis. In M. Chickering and J. Halpern, editors, Proceedings of the 20th Conference on Uncertainty in Artificial Intel ligence, pages 59--66, Banff, Alberta, Canada, 2004.

Digital Library

[5]

S. F. Chen and R. Rosenfeld. A Gaussian prior for smoothing maximum entropy models. Technical report, Carnegie Mellon University, CMU-CS-99-108, 1999.

[6]

S. C. Deerwester, S. T. Dumais, T. K. Landauer, G. W. Furnas, and R. A. Harshman. Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41(6):391--407, 1990.

[7]

E. Erosheva, S. Fienberg, and J. Lafferty. Mixed membership models of scientific publications. Proceedings of the National Academy of Sciences, 101(Suppl. 1), 2004.

[8]

P. Gehler, A. Holub, and M. Welling. The rate adapting Poisson model for information retrieval and object recognition. In Proceedings of the 23rd International Conference on Machine Learning, 2006.

Digital Library

[9]

T. Griffiths and M. Steyvers. Finding scientific topics. Proceedings of the National Academy of Sciences, 101(suppl. 1):5228--5235, 2004.

[10]

T. Griffiths, M. Steyvers, D. Blei, and J. Tenenbaum. Integrating topics and syntax. In Advances in Neural Information Processing Systems 17, Vancouver, British Columbia, Canada, 2004.

[11]

T. Hastie, R. Tibshirani, and J. H. Friedman. The Elements of Statistical Learning. Springer, 2001.

[12]

G. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14:1771--1800, 2002.

Digital Library

[13]

T. Hofmann. Probabilistic latent semantic analysis. In Proceedings of the 15th Conference on Uncertainty in Artificial Intel ligence, Stockholm, Sweden, 1999.

Digital Library

[14]

I. T. Jolliffe. Principal Component Analysis. Springer Verlag, 2002.

[15]

M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. In Proceedings of the NATO Advanced Study Institute on Learning in graphical models, pages 105--161, 1998.

Digital Library

[16]

C. Kemp, T. L. Griffiths, and J. Tenenbaum. Discovering latent classes in relational data. Technical report, MIT CSAIL, 2004.

[17]

A. McCallum, A. Corrada-Emanuel, and X. Wang. Topic and role discovery in social networks. In Proceedings of the 18th International Joint Conference on Artificial Intelligence, 2005.

Digital Library

[18]

A. McCallum, C. Pal, G. Druck, and X. Wang. Multi-conditional learning: Generative/discriminative training for clustering and classification. In Proceedings of the 21st National Conference on Artificial Intel ligence, 2006.

Digital Library

[19]

T. Minka and J. Lafferty. Expectation-propagation for the generative aspect model. In Proceedings of the 18th Conference on Uncertainty in Artificial Intelligence, 2002.

Digital Library

[20]

K. Nowicki and T. A. Snijders. Estimation and prediction for stochastic blockstructures. Journal of the American Statistical Association, 96(455), 2001.

[21]

S. Roweis and Z. Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2):305--345, 1999.

Digital Library

[22]

R. Salakhutdinov, A. Mnih, and G. Hinton. Restricted Boltzmann machines for collaborative filtering. In Proceedings of the 24th International Conference on Machine Learning, 2007.

Digital Library

[23]

G. Salton and M. McGill.Introduction to Modern Information Retrieval. McGraw-Hill, 1983.

Digital Library

[24]

M. Steyvers, P. Smyth, M. Rosen-Zvi, and T. Griffiths. Probabilistic author-topic models for information discovery. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2004.

Digital Library

[25]

Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. Hierarchical Dirichlet processes. Technical report, University of California, Berkeley, Department of Statistics, 2004.

[26]

M. E. Tipping and C. M. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society, Series B, 21(3):611--622, 1990.

[27]

X. Wang and A. McCallum. Topics over time: A non-Markov continuous-time model of topical trends. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2006.

Digital Library

[28]

X. Wang, N. Mohanty, and A. McCallum. Group and topic discovery from relations and their attributes. In Advances in Neural Information Processing Systems 18, Vancouver, British Columbia, Canada, 2005.

[29]

M. Welling, M. Rosen-Zvi, and G. Hinton. Exponential family harmoniums with an application to information retrieval. In Advances in Neural Information Processing Systems 17, Vancouver, British Columbia, Canada, 2004.

[30]

E. Xing, R. Yan, and A. G. Hauptmann. Mining associated text and images with dual--wing harmoniums. In Proceedings of the 21st Conference on Uncertainty in Artificial Intel ligence, 2005.

[31]

J. Yang, Y. Liu, E. P. Xing, and A. Hauptmann. Harmonium-based models for semantic video representation and classification. In Proceedings of the Seventh SIAM International Conference on Data Mining, 2007.

[32]

S. Yu, K. Yu, V. Tresp, H.-P. Kriegel, and M. Wu. Supervised probabilistic principal component analysis. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 464--473, New York, NY, USA, 2006. ACM Press.

Digital Library

[33]

C. Zhai and J. Lafferty. A study of smoothing methods for language models applied to information retrieval. ACM Transactions on Information System, 22(2):179--214, 2004.

Digital Library

Cited By

Shi XYu P(2012)Dimensionality Reduction on Heterogeneous Feature SpaceProceedings of the 2012 IEEE 12th International Conference on Data Mining10.1109/ICDM.2012.30(635-644)Online publication date: 10-Dec-2012
https://dl.acm.org/doi/10.1109/ICDM.2012.30
Gupta SPhung DAdams BTran TVenkatesh SRao BKrishnapuram BTomkins AYang Q(2010)Nonnegative shared subspace learning and its application to social media retrievalProceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining10.1145/1835804.1835951(1169-1178)Online publication date: 25-Jul-2010
https://dl.acm.org/doi/10.1145/1835804.1835951

Index Terms

Generalized component analysis for text with heterogeneous attributes
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

Generalized Principal Component Analysis (GPCA)

This paper presents an algebro-geometric solution to the problem of segmenting an unknown number of subspaces of unknown and varying dimensions from sample data points. We represent the subspaces with a set of homogeneous polynomials whose degree is the ...
Generating summary keywords for emails using topics
IUI '08: Proceedings of the 13th international conference on Intelligent user interfaces

Email summary keywords, used to concisely represent the gist of an email, can help users manage and prioritize large numbers of messages. We develop an unsupervised learning framework for selecting summary keywords from emails using latent ...
Supervised dimensionality reduction technology of generalized discriminant component analysis and its kernelization forms
Highlights
- Supervised subspace projection is a major method conducive to pattern recognition.
Abstract
Supervised subspace projection technology is a major method for dimensionality reduction in pattern recognition. At present, most supervised subspace projection algorithms are derived from the multi-dimensional extended version of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '07: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining

August 2007

1080 pages

ISBN:9781595936097

DOI:10.1145/1281192

General Chair:
Pavel Berkhin
Yahoo!, USA
,
Program Chairs:
Rich Caruana
Cornell University, USA
,
Xindong Wu
University of Vermont, USA

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

KDD07

Sponsor:

KDD07: The 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 12 - 15, 2007

California, San Jose, USA

Acceptance Rates

KDD '07 Paper Acceptance Rate 111 of 573 submissions, 19%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
689
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Shi XYu P(2012)Dimensionality Reduction on Heterogeneous Feature SpaceProceedings of the 2012 IEEE 12th International Conference on Data Mining10.1109/ICDM.2012.30(635-644)Online publication date: 10-Dec-2012
https://dl.acm.org/doi/10.1109/ICDM.2012.30
Gupta SPhung DAdams BTran TVenkatesh SRao BKrishnapuram BTomkins AYang Q(2010)Nonnegative shared subspace learning and its application to social media retrievalProceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining10.1145/1835804.1835951(1169-1178)Online publication date: 25-Jul-2010
https://dl.acm.org/doi/10.1145/1835804.1835951

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents