research-article

Learning with dual heterogeneity: a nonparametric bayes model

Authors:

Jingrui HeAuthors Info & Claims

KDD '14: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 582 - 590

https://doi.org/10.1145/2623330.2623727

Published: 24 August 2014 Publication History

Abstract

Traditional data mining techniques are designed to model a single type of heterogeneity, such as multi-task learning for modeling task heterogeneity, multi-view learning for modeling view heterogeneity, etc. Recently, a variety of real applications emerged, which exhibit dual heterogeneity, namely both task heterogeneity and view heterogeneity. Examples include insider threat detection across multiple organizations, web image classification in different domains, etc. Existing methods for addressing such problems typically assume that multiple tasks are equally related and multiple views are equally consistent, which limits their application in complex settings with varying task relatedness and view consistency. In this paper, we advance state-of-the-art techniques by adaptively modeling task relatedness and view consistency via a nonparametric Bayes model: we model task relatedness using normal penalty with sparse covariances, and view consistency using matrix Dirichlet process. Based on this model, we propose the NOBLE algorithm using an efficient Gibbs sampler. Experimental results on multiple real data sets demonstrate the effectiveness of the proposed algorithm.

Supplementary Material

MP4 File (p582-sidebyside.mp4)

Download
299.61 MB

References

[1]

R. Ando and T. Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817--1853, 2005.

Digital Library

[2]

C. Antoniak. Mixtures of dirichlet processes with applications to bayeisan nonparametric problems. The Annals of Statistics, 2:1152--1174, 1974.

[3]

C. Archambeau, S. Guo, and O. Zoeter. Sparse bayesian multi-task learning. NIPS, 2011.

[4]

A. Asuncion and D. Newman. UCI machine learning repository, 2007.

[5]

B. Bakker and T. Hesks. Task clustering and gating for bayesian multitask learning. JMLR, 4:83--99, 2003.

Digital Library

[6]

D. Blackwell and J. MacQueen. Ferguson distributions via polya urn schemes. The Annals of Statistics, 1:353--355, 1973.

[7]

D. M. Blei, T. L. Griffiths, M. I. Jordan, and J. B. Tenenbaum. Hierarchical topic models and the nested chinese restaurant process. In NIPS, 2003.

Digital Library

[8]

A. Blum and T. M. Mitchell. Combining labeled and unlabeled sata with co-training. In COLT, 1998.

Digital Library

[9]

D. Burr and H. Doss. A bayesian semiparametric model for random-effects meta-analysis. Journal of the American Statistical Association, 100(469):242--251, 2005.

[10]

J. Chen, T. Lei, J. Liu, and J. Ye. A convex formulation for learning shared structures from multiple tasks. ICML, 2009.

Digital Library

[11]

J. Chen, J. Liu, and J. Ye. Learning incoherent sparse and low-rank patterns from multiple tasks. In KDD, pages 1179--1188, 2010.

Digital Library

[12]

J. Chen, J. Zhou, and J. Ye. Integrating low-rank and group-sparse structures for robust multi-task learning. In KDD, pages 42--50, 2011.

Digital Library

[13]

C. Christoudias, R. Urtasun, and T. Darrell. Multi-view learning in the presence of view disagreement. In UAI, pages 88--96, 2008.

[14]

L. Ding, A. Yilmaz, and R. Yan. Interactive image segmentation using dirichlet process multiple-view learning. IEEE Transactions on Image Processing, 21(4):2119--2129, 2012.

Digital Library

[15]

D. Dunson, Y. Xue, and L. Carin. The matrix stick- breaking process: exible bayes meta analysis. Journal of the American Statistical Association, 103:317--327, 2008.

[16]

J. Farquhar, D. Hardoon, H. Meng, J. Shawe-Taylor, and S. Szedmak. Two view learning: Svm-2k, theory and practice. NIPS, 2005.

[17]

T. Ferguson. A bayesian analysis of some nonparametric problems. The Annals of Statistics, 1:209--230, 1973.

[18]

H. Gavert, J. Hurri, J. Sarela, and A. Hyvarinen. The fastica package for matlab. http://research.ics.aalto.fi/ica/fastica/, 2005.

[19]

S. Han, X. Liao, and L. Carin. Cross-domain multitask learning with latent probit models. NIPS, 2012.

[20]

M. Harel and S. Mannor. Learning from multiple outlooks. In ICML, pages 401--408, 2011.

Digital Library

[21]

J. He and R. Lawrence. A graphbased framework for multi-task multi-view learning. In ICML, pages 25--32, 2011.

[22]

J. He, Y. Liu, and Q. Yang. Linking heterogeneous input spaces with pivots for multi-task learning. In SDM, 2014.

[23]

X. Jin, F. Zhuang, S. Wang, Q. He, and Z. Shi. Shared structure learning for multiple tasks with multiple views. In ECML/PKDD (2), pages 353--368, 2013.

[24]

S. M. Kakade and D. P. Foster. Multi-view regression via canonical correlation analysis. In COLT, pages 82--96, 2007.

Digital Library

[25]

A. Kumar, P. Rai, and H. D. III. Co-regularized multi-view spectral clustering. In NIPS, pages 1413--1421, 2011.

Digital Library

[26]

Y.-X. Li, S. Ji, S. Kumar, J. Ye, and Z.-H. Zhou. Drosophila gene expression pattern annotation through multi-instance multi-label learning. IEEE/ACM Trans. Comput. Biology Bioinform., 9(1):98--112, 2012.

Digital Library

[27]

Q. Liu, X. Liao, and L. Carin. Semi-supervised multi-task leraning. NIPS, 2007.

[28]

I. Muslea, S. Minton, and C. A. Knoblock. Active + semi-supervised learning = robust multi-view learning. In ICML, pages 435--442, 2002.

Digital Library

[29]

K. Nigam and R. Ghani. Analyzing the effectiveness and applicability of co-training. In CIKM, pages 86--93, 2000.

Digital Library

[30]

J. O'Sullivan and S. Thrun. Discovering structure in multiple learning tasks: The tc algorithm. ICML, pages 489--497, 1996.

[31]

M. Tipping and C. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society, Series B, 61:611--622, 1999.

[32]

L. van der Maaten, E. Postma, and H. van den Herik. Dimensionality reduction: A comparative review. Tilburg University Technical Report, TiCC-TR 2009-005, 2009.

[33]

H. Wang, F. Nie, H. Huang, S. L. Risacher, A. J. Saykin, and L. Shen. Identifying disease sensitive and quantitative trait-relevant biomarkers from multidimensional heterogeneous imaging genetics data via sparse multimodal multitask learning. Bioinformatics, 28(12):127--136, 2012.

Digital Library

[34]

W. Wang and Z.-H. Zhou. A new analysis of co-training. In ICML, pages 1135--1142, 2010.

Digital Library

[35]

Y. Xue, X. Liao, L. Carin, and B. Krishnapuram. Multi-task learning for classification with dirichlet process priors. Journal of Machine Learning Research, 8:35--63, 2007.

Digital Library

[36]

C. Yau, O. Papaspiliopoulos, G. Roberts, and C. Holmes. Nonparametric hidden markov models with application to the analysis of copy-number-variation in mammalian genomes. Journal of Royal Statistical Society: Series B, 73(1):37--57, 2010.

[37]

K. Yu and W. Chu. Gaussian process models for link analysis and transfer learning. In NIPS, 2007.

[38]

K. Yu, A. Schwaighofer, and V. Tresp. Learning gaussian processes from multiple tasks. ICML, 2005.

Digital Library

[39]

K. Yu, A. Schwaighofer, V. Tresp, W. Ma, and H. Zhang. Collaborative ensemble learning: Combining collaborative and content-based information filtering via hierarchical bayes. UAI, 2003.

Digital Library

[40]

K. Yu, V. Tresp, and S. Yu. A nonparametric hierarchical bayesian framework for information filtering. Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2004.

Digital Library

[41]

D. Zhang, J. He, and R. D. Lawrence. Mi2ls: multi-instance learning from multiple informationsources. In KDD, pages 149--157, 2013.

Digital Library

[42]

D. Zhang, J. He, Y. Liu, L. Si, and R. D. Lawrence. Multi-view transfer learning with a large margin approach. In KDD, pages 1208--1216, 2011.

Digital Library

[43]

J. Zhang and J. Huan. Inductive multi-task learning with multiple view data. In KDD, pages 543--551, 2012.

Digital Library

[44]

Y. Zhang and D.-Y. Yeung. A convex formulation for learning task relationships in multi-task learning. CoRR, abs/1203.3536, 2012.

[45]

J. Zhou, J. Chen, and J. Ye. Clustered multi-task learning via alternating structure optimization. In NIPS, pages 702--710, 2011.

Digital Library

Cited By

Li XHuan JCuzzocrea AAllan JPaton NSrivastava DAgrawal RBroder AZaki MCandan SLabrinidis ASchuster AWang H(2018)Interactions Modeling in Multi-Task Multi-View Learning with Consistent Task DiversityProceedings of the 27th ACM International Conference on Information and Knowledge Management10.1145/3269206.3271670(853-861)Online publication date: 17-Oct-2018
https://dl.acm.org/doi/10.1145/3269206.3271670
Yang PDavulcu HZhu YHe J(2018)A Generalized Hierarchical Multi-Latent Space Model for Heterogeneous LearningIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2016.261151428:12(3154-3168)Online publication date: 31-Dec-2018
https://dl.acm.org/doi/10.1109/TKDE.2016.2611514
(2018)Semantic Feature Learning for Heterogeneous Multitask Classification via Non-Negative Matrix FactorizationIEEE Transactions on Cybernetics10.1109/TCYB.2017.273281848:8(2284-2293)Online publication date: Aug-2018
https://doi.org/10.1109/TCYB.2017.2732818
Show More Cited By

Index Terms

Learning with dual heterogeneity: a nonparametric bayes model
1. Computing methodologies
  1. Machine learning
2. Mathematics of computing
  1. Probability and statistics
    1. Nonparametric statistics

Recommendations

Asymptotic Bayesian structure learning using graph supports for Gaussian graphical models

The theory of Gaussian graphical models is a powerful tool for independence analysis between continuous variables. In this framework, various methods have been conceived to infer independence relations from data samples. However, most of them result in ...
Likelihood-free approximate Gibbs sampling
Abstract
Likelihood-free methods such as approximate Bayesian computation (ABC) have extended the reach of statistical inference to problems with computationally intractable likelihoods. Such approaches perform well for small-to-moderate dimensional ...
A Bayesian model selection method with applications

In this paper, we consider Bayesian model selection using the well-known Bayes factor. A method on the basis of path sampling for computing the ratio of two normalizing constants involved in the Bayes factor is proposed. The key idea is to construct a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '14: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining

August 2014

2028 pages

ISBN:9781450329569

DOI:10.1145/2623330

General Chairs:
Sofus Macskassy
Facebook
,
Claudia Perlich
Dstillery
,
Program Chairs:
Jure Leskovec
Stanford University
,
Wei Wang
UCLA
,
Rayid Ghani
University of Chicago

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '14

Sponsor:

KDD '14: The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 24 - 27, 2014

New York, New York, USA

Acceptance Rates

KDD '14 Paper Acceptance Rate 151 of 1,036 submissions, 15%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
733
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li XHuan JCuzzocrea AAllan JPaton NSrivastava DAgrawal RBroder AZaki MCandan SLabrinidis ASchuster AWang H(2018)Interactions Modeling in Multi-Task Multi-View Learning with Consistent Task DiversityProceedings of the 27th ACM International Conference on Information and Knowledge Management10.1145/3269206.3271670(853-861)Online publication date: 17-Oct-2018
https://dl.acm.org/doi/10.1145/3269206.3271670
Yang PDavulcu HZhu YHe J(2018)A Generalized Hierarchical Multi-Latent Space Model for Heterogeneous LearningIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2016.261151428:12(3154-3168)Online publication date: 31-Dec-2018
https://dl.acm.org/doi/10.1109/TKDE.2016.2611514
(2018)Semantic Feature Learning for Heterogeneous Multitask Classification via Non-Negative Matrix FactorizationIEEE Transactions on Cybernetics10.1109/TCYB.2017.273281848:8(2284-2293)Online publication date: Aug-2018
https://doi.org/10.1109/TCYB.2017.2732818
Yang PTan QZhu YHe J(2018)Heterogeneous representation learning with separable structured sparsity regularizationKnowledge and Information Systems10.1007/s10115-017-1094-555:3(671-694)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s10115-017-1094-5
Liu BXu ZDai BBai HFang XRen YZhe S(2017)Learning from semantically dependent multi-tasks2017 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN.2017.7966296(3498-3505)Online publication date: May-2017
https://doi.org/10.1109/IJCNN.2017.7966296
Li XChandrasekaran SHuan J(2017)Lifelong multi-task multi-view learning using latent spaces2017 IEEE International Conference on Big Data (Big Data)10.1109/BigData.2017.8257909(37-46)Online publication date: Dec-2017
https://doi.org/10.1109/BigData.2017.8257909
Li XHuan JMukhopadhyay SZhai CBertino ECrestani FMostafa JTang JSi LZhou XChang YLi YSondhi P(2016)aptMTVLProceedings of the 25th ACM International on Conference on Information and Knowledge Management10.1145/2983323.2983783(1171-1180)Online publication date: 24-Oct-2016
https://dl.acm.org/doi/10.1145/2983323.2983783
Wei YZheng YYang QKrishnapuram BShah MSmola AAggarwal CShen DRastogi R(2016)Transfer Knowledge between CitiesProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining10.1145/2939672.2939830(1905-1914)Online publication date: 13-Aug-2016
https://dl.acm.org/doi/10.1145/2939672.2939830
Yang PYang HFu HZhou DYe JLappas THe J(2016)Jointly Modeling Label and Feature Heterogeneity in Medical InformaticsACM Transactions on Knowledge Discovery from Data10.1145/276883110:4(1-25)Online publication date: 11-May-2016
https://dl.acm.org/doi/10.1145/2768831
Yang PHe J(2016)Heterogeneous Representation Learning with Structured Sparsity Regularization2016 IEEE 16th International Conference on Data Mining (ICDM)10.1109/ICDM.2016.0065(539-548)Online publication date: Dec-2016
https://doi.org/10.1109/ICDM.2016.0065
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents