Cited By
View all- Liu JTalwar KCharikar MCohen E(2019)Private selection from private candidatesProceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing10.1145/3313276.3316377(298-309)Online publication date: 23-Jun-2019
Supervising unsupervised learning
Authors: 
Vikas K. Garg, 
Adam KalaiAuthors Info & Claims
Vikas K. Garg, Adam KalaiAuthors Info & ClaimsPages 4996 - 5006
Abstract
We introduce a framework to transfer knowledge acquired from a repository of (heterogeneous) supervised datasets to new unsupervised datasets. Our perspective avoids the subjectivity inherent in unsupervised learning by reducing it to supervised learning, and provides a principled way to evaluate unsupervised algorithms. We demonstrate the versatility of our framework via rigorous agnostic bounds on a variety of unsupervised problems. In the context of clustering, our approach helps choose the number of clusters and the clustering algorithm, remove the outliers, and provably circumvent Kleinberg's impossibility result. Experiments across hundreds of problems demonstrate improvements in performance on unsupervised data with simple algorithms despite the fact our problems come from heterogeneous domains. Additionally, our framework lets us leverage deep networks to learn common features across many small datasets, and perform zero shot learning.
References
[1]
Jon Kleinberg. An impossibility theorem for clustering. In Advances in neural information processing systems (NIPS), pages 463-470, 2003.
[2]
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering (TKDE), 22:1345-1359, 2010.
[3]
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. Semi-supervised learning with deep generative models. In NIPS, 2014.
[4]
N. Siddharth, Brooks Paige, Jan-Willem van de Meent, Alban Desmaison, Noah D. Goodman, Pushmeet Kohli, Frank Wood, and Philip H.S. Torr. Learning disentangled representations with semi-supervised deep generative models. In NIPS, 2017.
[5]
Marcus Rohrbach, Sandra Ebert, and Bernt Schiele. Transfer learning in a transductive setting. In NIPS, 2013.
[6]
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015.
[7]
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Unsupervised domain adaptation with residual transfer networks. In NIPS, 2016.
[8]
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In NIPS, 2016.
[9]
Michael J Kearns, Robert E Schapire, and Linda M Sellie. Toward efficient agnostic learning. Machine Learning, 17(2-3):115-141, 1994.
[10]
Maria-Florina Balcan, Vaishnavh Nagarajan, Ellen Vitercik, and Colin White. Learning-theoretic foundations of algorithm configuration for combinatorial partitioning problems. In Conference on Learning Theory, pages 213-274, 2017.
[11]
Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 847-855. ACM, 2013.
[12]
Nicolo Fusi, Rishit Sheth, and Huseyn Melih Elibol. Probabilistic matrix factorization for automated machine learning. 2018.
[13]
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
[14]
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In NIPS, pages 2951-2959, 2012.
[15]
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. In The biology and technology of intelligent autonomous agents, pages 165-196. Springer, 1995.
[16]
Maria-Florina Balcan, Avrim Blum, and Santosh Vempala. Efficient representations for lifelong learning and autoencoding. In Workshop on Computational Learning Theory (COLT), 2015.
[17]
Reza Bosagh Zadeh and Shai Ben-David. A uniqueness theorem for clustering. In Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence (UAI), pages 639-646. AUAI Press, 2009.
[18]
Margareta Ackerman and Shai Ben-David. Measures of clustering quality: A working set of axioms for clustering. In NIPS, 2008.
[19]
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In ICML, 2015.
[20]
S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016.
[21]
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In ICCV, 2017.
[22]
Chirag Gupta, Arun Sai Suggala, Ankit Goyal, Harsha Vardhan Simhadri, Bhargavi Paranjape, Ashish Kumar, Saurabh Goyal, Raghavendra Udupa, Manik Varma, and Prateek Jain. ProtoNN: Compressed and accurate kNN for resource-scarce devices. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 1331-1340, 2017.
[23]
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2(1):193-218, 1985.
[24]
Peter J Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
[25]
G. C. Tseng. Penalized and weighted k-means for clustering with scattered objects and prior information in high-throughput biological data. Bioinformatics, 23:2247-2255, 2007.
[26]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252, 2015.
[27]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research (JMLR), 12:2825-2830, 2011.
[28]
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
- Supervising unsupervised learning
Recommendations
Supervising an Unsupervised Neural Network
ACIIDS '09: Proceedings of the 2009 First Asian Conference on Intelligent Information and Database SystemsMachine learning is the field that is dedicated to the design and development of algorithms and techniques that allow computers to “learn”. Two common types of learning that are often mentioned are supervised learning and unsupervised learning. One ...
Unsupervised Selective Labeling for More Effective Semi-supervised Learning
Computer Vision – ECCV 2022AbstractGiven an unlabeled dataset and an annotation budget, we study how to selectively label a fixed number of instances so that semi-supervised learning (SSL) on such a partially labeled dataset is most effective. We focus on selecting the right data ...
Unsupervised meta-learning for few-shot learning
Highlights- Unsupervised meta-learning that auto-constructs tasks from unlabeled data.
- ...
AbstractMeta-learning is an effective tool to address the few-shot learning problem, which requires new data to be classified considering only a few training examples. However, when used for classification, it requires large labeled datasets, ...
Comments
Information & Contributors
Information
Published In
December 2018
11021 pages
Publisher
Curran Associates Inc.
Red Hook, NY, United States
Publication History
Published: 03 December 2018
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 51Total Downloads
- Downloads (Last 12 months)9
- Downloads (Last 6 weeks)2
Reflects downloads up to 15 Jan 2025
Other Metrics
Citations
Cited By
View all- Liu JTalwar KCharikar MCohen E(2019)Private selection from private candidatesProceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing10.1145/3313276.3316377(298-309)Online publication date: 23-Jun-2019
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in