research-article

Multi-task Sparse Structure Learning

Authors:

Andre R. Goncalves,

Soumyadeep Chatterjee,

Vidyashankar Sivakumar,

Fernando J. Von Zuben,

Arindam BanerjeeAuthors Info & Claims

CIKM '14: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management

Pages 451 - 460

https://doi.org/10.1145/2661829.2662091

Published: 03 November 2014 Publication History

Abstract

Multi-task learning (MTL) aims to improve generalization performance by learning multiple related tasks simultaneously. While sometimes the underlying task relationship structure is known, often the structure needs to be estimated from data at hand. In this paper, we present a novel family of models for MTL, applicable to regression and classification problems, capable of learning the structure of task relationships. In particular, we consider a joint estimation problem of the task relationship structure and the individual task parameters, which is solved using alternating minimization. The task relationship structure learning component builds on recent advances in structure learning of Gaussian graphical models based on sparse estimators of the precision (inverse covariance) matrix. We illustrate the effectiveness of the proposed model on a variety of synthetic and benchmark datasets for regression and classification. We also consider the problem of combining climate model outputs for better projections of future climate, with focus on temperature in South America, and show that the proposed model outperforms several existing methods for the problem.

References

[1]

IPCC Fifth assessment report. Intergovernmental Panel on Climate Change, 2013.

[2]

J. Abernethy, F. Bach, T. Evgeniou, and J.-P. Vert. Low-rank matrix factorization with attributes. CoRR, abs/cs/0611124, 2006.

[3]

A. Argyriou, T. Evgeniou, and M. Pontil. Multi-task feature learning. In NIPS, 2007.

Digital Library

[4]

B. Bakker and T. Heskes. Task clustering and gating for bayesian multitask learning. JMLR, 4:83--99, 2003.

Digital Library

[5]

O. Banerjee, L. El Ghaoui, and A. d'Aspremont. Model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data. JMLR, 9:485--516, 2008.

Digital Library

[6]

A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183--202, 2009.

Digital Library

[7]

M. Bentsen et al. The Norwegian Earth System Model, NorESM1-M-Part 1: Description and basic evaluation. Geo. Model Dev. Disc., 5:2843--2931, 2012.

[8]

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn., 3(1):1--122, 2011.

Digital Library

[9]

V. Brovkin, L. Boysen, T. Raddatz, V. Gayler, A. Loew, and M. Claussen. Evaluation of vegetation cover and land-surface albedo in MPI-ESM CMIP5 simulations. J. of Adv. in Modeling Earth Sys., 2013.

[10]

R. Caruana. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning, pages 41{48. Morgan Kaufmann, 1993.

[11]

W. Collins et al. Development and evaluation of an Earth-system model--HadGEM2. Geosci. Model Dev. Discuss, 4:997--1062, 2011.

[12]

A. P. Dempster. Covariance selection. Biometrics, pages 157--175, 1972.

[13]

J. Dufresne et al. Climate change projections using the IPSL-CM5 Earth System Model: from CMIP3 to CMIP5. Climate Dynamics, 2012.

[14]

T. Evgeniou, C. A. Micchelli, and M. Pontil. Learning multiple tasks with kernel methods. JMLR, 6, 2005.

Digital Library

[15]

T. Evgeniou and M. Pontil. Regularized multi{task learning. In KDD, pages 109--117, 2004.

Digital Library

[16]

J. Friedman, T. Hastie, and R. Tibshiran. Sparse inverse covariance estimation with the graphical lasso. Biostatistics, 9 3:432--441, 2008.

[17]

H. B. Gordon et al. The CSIRO Mk3 climate system model, volume 130. CSIRO Atmospheric Research, 2002.

[18]

A. Gunawardana and W. Byrne. Convergence theorems for generalized alternating minimization procedures. JMLR, 6:2049--2073, 2005.

Digital Library

[19]

T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning; Data mining, Inference and Prediction. Springer Verlag, 2001.

[20]

X. He, D. Cai, and P. Niyogi. Laplacian score for feature selection. In NIPS, pages 507--514. 2006.

Digital Library

[21]

C. Hsieh, I. Dhillon, P. Ravikumar, and A. Banerjee. A Divide-and-Conquer Method for Sparse Inverse Covariance Estimation. In NIPS, 2012.

[22]

A. Jalali, P. Ravikumar, S. Sanghavi, and C. Ruan. A Dirty Model for Multi-task Learning. NIPS, 2010.

Digital Library

[23]

S. Ji and J. Ye. An Accelerated Gradient Method for Trace Norm Minimization. In ICML, 2009.

Digital Library

[24]

S. Kim and E. P. Xing. Tree-Guided Group Lasso for MultiTask Regression with Structured Sparsity. In ICML, pages 543--550, 2010.

Digital Library

[25]

S. L. Lauritzen. Graphical Models. Oxford University Press, Oxford, 1996.

[26]

H. Liu, F. Han, M. Yuan, J. Lafferty, and L. Wasserman. High Dimensional Semiparametric Gaussian Copula Graphical Models. The Annals ofStatistics, 40(40):2293--2326, 2012.

[27]

K. V. Mardia and R. Marshall. Maximum likelihood estimation of models for residual covariance in spatial regression. Biometrika, 71(1):135--146, 1984.

[28]

N. Meinshausen and P. Buhlmann. High-dimensional graphs and variable selection with the lasso. Annals ofStatistics, 34 3:1436--1462, 2006.

[29]

N. Meinshausen and P. Buhlmann. Stability selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(4):417--473, 2010.

[30]

J. A. Nelder and R. Baker. Generalized linear models. Wiley Online Library, 1972.

[31]

G. Obozinski, B. Taskar, and M. I. Jordan. Joint covariate selection and joint subspace selection for multiple classification problems. Statistics and Computing, 20:231--252, 2010.

Digital Library

[32]

V. A. Ramos. South America. In Encyclopaedia Britannica Online Academic Edition. 2014.

[33]

K. Subbian and A. Banerjee. Climate Multi-model Regression using Spatial Smoothing. In SDM, 2013.

[34]

Z. M. Subin, L. N. Murphy, F. Li, C. Bonfils, and W. J. Riley. Boreal lakes moderate seasonal and diurnal temperature variation and perturb atmospheric circulation: analyses in the Community Earth System Model 1 (CESM1). Tellus A, 64, 2012.

[35]

K. E. Taylor, R. J. Stouffer, and G. A. Meehl. An overview of CMIP5 and the experiment design. Bull. of the Am. Met. Soc., 93(4):485, 2012.

[36]

H. Wang, A. Banerjee, C. jui Hsieh, P. Ravikumar, and I. Dhillon. Large scale distributed sparse precision estimation. In NIPS, pages 584--592, 2013.

[37]

X. Wang, C. Zhang, and Z. Zhang. Boosted multi-task learning for face verification with applications to web image and video search. CVPR, pages 142--149, 2009.

[38]

W. Washington et al. The use of the Climate-science Computational End Station (CCES) development and grand challenge team for the next IPCC assessment: an operational plan. J. of Physics, 125(1), 2008.

[39]

M. Watanabe et al. Improved climate simulation by MIROC5: Mean states, variability, and climate sensitivity. J. of Clim., 23:6312--6335, 2010.

[40]

C. Widmer and G. Ratsch. Multitask learning in computational biology. JMLR - Proceedings Track, 27:207--216, 2012.

[41]

Y. Xue, X. Liao, L. Carin, and B. Krishnapuram. Multi-task learning for classification with Dirichlet process priors. JMLR, 8:35--63, 2007.

Digital Library

[42]

S. Yukimoto, Y. Adachi, and M. Hosaka. A New Global Climate Model of the Meteorological Research Institute: MRI-CGCM3: Model Description and Basic Performance. J. of the Met. Soc. of Japan, 90:23--64, 2012.

[43]

L. Zhang, T. Wu, X. Xin, M. Dong, and Z. Wang. Projections of annual mean air temperature and precipitation over the globe and in China during the 21st century by the BCC Climate System Model BCC CSM1. 0. Acta Met. Sinica, 26(3):362--375, 2012.

[44]

Y. Zhang and D.-Y. Yeung. A convex formulation for learning task relationships in multi-task learning. In UAI, 2010.

Digital Library

[45]

J. Zhou, J. Chen, and J. Ye. Clustered Multi-Task learning via alternating structure optimization. In NIPS, 2011.

Digital Library

Cited By

Lin JChen QXue BZhang M(2024)Evolutionary Multitasking for Multiobjective Feature Selection in ClassificationIEEE Transactions on Evolutionary Computation10.1109/TEVC.2023.333874028:6(1852-1866)Online publication date: Dec-2024
https://doi.org/10.1109/TEVC.2023.3338740
Chang XZhou MYang YYang P(2024)Adaptive Prior Correction in Alzheimer’s Disease Spatio-Temporal Modeling via Multi-task LearningInternet of Things of Big Data for Healthcare10.1007/978-3-031-52216-1_6(69-83)Online publication date: 29-Jan-2024
https://doi.org/10.1007/978-3-031-52216-1_6
Li YYan HJin R(2023)Multi-Task Learning With Latent Variation Decomposition for Multivariate Responses in a Manufacturing NetworkIEEE Transactions on Automation Science and Engineering10.1109/TASE.2022.314897720:1(285-295)Online publication date: Jan-2023
https://doi.org/10.1109/TASE.2022.3148977
Show More Cited By

Index Terms

Multi-task Sparse Structure Learning
1. Computing methodologies
  1. Machine learning
    1. Learning settings
2. Mathematics of computing
  1. Mathematical software

Recommendations

Multi-task sparse structure learning with Gaussian copula models

Multi-task learning (MTL) aims to improve generalization performance by learning multiple related tasks simultaneously. While sometimes the underlying task relationship structure is known, often the structure needs to be estimated from data at hand. In ...
A Regularization Approach to Learning Task Relationships in Multitask Learning

Multitask learning is a learning paradigm that seeks to improve the generalization performance of a learning task with the help of some other related tasks. In this article, we propose a regularization approach to learning the relationships between ...
Learning Task Grouping using Supervised Task Space Partitioning in Lifelong Multitask Learning
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

Lifelong multitask learning is a multitask learning framework in which a learning agent faces the tasks that need to be learnt in an online manner. Lifelong multitask learning framework may be applied to a variety of applications such as image ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '14: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management

November 2014

2152 pages

ISBN:9781450325981

DOI:10.1145/2661829

General Chairs:
Jianzhong Li
Harbin Inst. of Technology
,
X. Sean Wang
Fudan University
,
Program Chairs:
Minos Garofalakis
Technical University of Crete, Greece
,
Ian Soboroff
National Institute of Standards, USA
,
Torsten Suel
New York University, USA
,
Min Wang
Google Research, USA

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

CIKM '14

Sponsor:

CIKM '14: 2014 ACM Conference on Information and Knowledge Management

November 3 - 7, 2014

Shanghai, China

Acceptance Rates

CIKM '14 Paper Acceptance Rate 175 of 838 submissions, 21%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
393
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lin JChen QXue BZhang M(2024)Evolutionary Multitasking for Multiobjective Feature Selection in ClassificationIEEE Transactions on Evolutionary Computation10.1109/TEVC.2023.333874028:6(1852-1866)Online publication date: Dec-2024
https://doi.org/10.1109/TEVC.2023.3338740
Chang XZhou MYang YYang P(2024)Adaptive Prior Correction in Alzheimer’s Disease Spatio-Temporal Modeling via Multi-task LearningInternet of Things of Big Data for Healthcare10.1007/978-3-031-52216-1_6(69-83)Online publication date: 29-Jan-2024
https://doi.org/10.1007/978-3-031-52216-1_6
Li YYan HJin R(2023)Multi-Task Learning With Latent Variation Decomposition for Multivariate Responses in a Manufacturing NetworkIEEE Transactions on Automation Science and Engineering10.1109/TASE.2022.314897720:1(285-295)Online publication date: Jan-2023
https://doi.org/10.1109/TASE.2022.3148977
Liu GZhang RHang RGe LShi CLiu Q(2023)Statistical Downscaling of Temperature Distributions in Southwest China by Using Terrain-Guided Attention NetworkIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing10.1109/JSTARS.2023.323910916(1678-1690)Online publication date: 2023
https://doi.org/10.1109/JSTARS.2023.3239109
Lin JChen QXue BZhang M(2023)AMTEA-Based Multi-task Optimisation for Multi-objective Feature Selection in ClassificationApplications of Evolutionary Computation10.1007/978-3-031-30229-9_40(623-639)Online publication date: 12-Apr-2023
https://dl.acm.org/doi/10.1007/978-3-031-30229-9_40
Li ZYan HTsung FZhang K(2022)Profile Decomposition Based Hybrid Transfer Learning for Cold-Start Data Anomaly DetectionACM Transactions on Knowledge Discovery from Data10.1145/353099016:6(1-28)Online publication date: 30-Jul-2022
https://dl.acm.org/doi/10.1145/3530990
Lin JChen QXue BZhang MWagner M(2022)Multi-task optimisation for multi-objective feature selection in classificationProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3520304.3528903(264-267)Online publication date: 9-Jul-2022
https://dl.acm.org/doi/10.1145/3520304.3528903
Price BMolstad ASherwood B(2021)Estimating Multiple Precision Matrices With Cluster Fusion RegularizationJournal of Computational and Graphical Statistics10.1080/10618600.2021.187496330:4(823-834)Online publication date: 19-Mar-2021
https://doi.org/10.1080/10618600.2021.1874963
Alesiani FYu SShaker AYin W(2020)Towards Interpretable Multi-task Learning Using Bilevel ProgrammingMachine Learning and Knowledge Discovery in Databases10.1007/978-3-030-67661-2_35(593-608)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.1007/978-3-030-67661-2_35
Liu YChen JGanguly ADy JTeredesai AKumar VLi YRosales RTerzi EKarypis G(2019)Nonparametric Mixture of Sparse Regressions on Spatio-Temporal Data -- An Application to Climate PredictionProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining10.1145/3292500.3330692(2556-2564)Online publication date: 25-Jul-2019
https://dl.acm.org/doi/10.1145/3292500.3330692
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents