research-article

Public Access

Simultaneous Private Learning of Multiple Concepts

Authors:

Uri StemmerAuthors Info & Claims

ITCS '16: Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science

Pages 369 - 380

https://doi.org/10.1145/2840728.2840747

Published: 14 January 2016 Publication History

Abstract

We investigate the {\em direct-sum} problem in the context of differentially private PAC learning: What is the sample complexity of solving k learning tasks simultaneously under differential privacy, and how does this cost compare to that of solving k learning tasks without privacy? In our setting, an individual example consists of a domain element x labeled by k unknown concepts (c₁,...,c_k). The goal of a multi-learner is to output k hypotheses (h₁,...,h_k) that generalize the input examples.

Without concern for privacy, the sample complexity needed to simultaneously learn $k$ concepts is essentially the same as needed for learning a single concept. Under differential privacy, the basic strategy of learning each hypothesis independently yields sample complexity that grows polynomially with k. For some concept classes, we give multi-learners that require fewer samples than the basic strategy. Unfortunately, however, we also give lower bounds showing that even for very simple concept classes, the sample cost of private multi-learning must grow polynomially in k.

References

[1]

R. Bassily, A. Smith, and A. Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In FOCS, pages 464--473, 2014.

Digital Library

[2]

A. Beimel, H. Brenner, S. P. Kasiviswanathan, and K. Nissim. Bounds on the sample complexity for private learning and private data release. Machine Learning, 94(3):401--437, 2014.

Digital Library

[3]

A. Beimel, S. P. Kasiviswanathan, and K. Nissim. Bounds on the sample complexity for private learning and private data release. In TCC, pages 437--454, 2010.

Digital Library

[4]

A. Beimel, K. Nissim, and U. Stemmer. Characterizing the sample complexity of private learners. In ITCS, pages 97--110, 2013.

Digital Library

[5]

A. Beimel, K. Nissim, and U. Stemmer. Private learning and sanitization: Pure vs. approximate differential privacy. In APPROX-RANDOM, pages 363--378, 2013.

[6]

A. Beimel, K. Nissim, and U. Stemmer. Learning privately with labeled and unlabeled examples. In SODA, pages 461--477, 2015.

Digital Library

[7]

A. Blum, C. Dwork, F. McSherry, and K. Nissim. Practical privacy: the SuLQ framework. In PODS, pages 128--138, 2005.

Digital Library

[8]

A. Blum, K. Ligett, and A. Roth. A learning theory approach to noninteractive database privacy. J. ACM, 60(2):12, 2013.

Digital Library

[9]

D. Boneh and J. Shaw. Collusion-secure fingerprinting for digital data. IEEE Transactions on Information Theory, 44(5):1897--1905, 1998.

Digital Library

[10]

M. Bun, K. Nissim, U. Stemmer, and S. Vadhan. Differentially private release and learning of threshold functions. In Proceedings of the 56th Annual IEEE Symposium on Foundations of Computer Science (FOCS 2015), pages 634--649, Berkeley, CA, USA, October 18--20, 2015.

Digital Library

[11]

M. Bun, J. Ullman, and S. P. Vadhan. Fingerprinting codes and the price of approximate differential privacy. In STOC, pages 1--10, 2014.

Digital Library

[12]

K. Chaudhuri and D. Hsu. Sample complexity bounds for differentially private learning. In S. M. Kakade and U. von Luxburg, editors, COLT, volume 19 of JMLR Proceedings, pages 155--186. JMLR.org, 2011.

[13]

C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT, pages 486--503, 2006.

Digital Library

[14]

C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In TCC, pages 265--284, 2006.

Digital Library

[15]

C. Dwork, G. N. Rothblum, and S. P. Vadhan. Boosting and differential privacy. In FOCS, pages 51--60, 2010.

Digital Library

[16]

C. Dwork, K. Talwar, A. Thakurta, and L. Zhang. Analyze gauss: Optimal bounds for privacy-preserving principal component analysis. In Proceedings of the 46th Annual ACM Symposium on Theory of Computing, STOC '14, pages 11--20, New York, NY, USA, 2014. ACM.

Digital Library

[17]

D. Feldman, A. Fiat, H. Kaplan, and K. Nissim. Private coresets. In M. Mitzenmacher, editor, Proceedings of the 41st Annual ACM Symposium on Theory of Computing, STOC 2009, Bethesda, MD, USA, May 31 - June 2, 2009, pages 361--370. ACM, 2009.

Digital Library

[18]

V. Feldman and D. Xiao. Sample complexity bounds on differentially private learning via communication complexity. In COLT, pages 1000--1019, 2014.

[19]

A. Gupta, M. Hardt, A. Roth, and J. Ullman. Privately releasing conjunctions and the statistical query barrier. In STOC, pages 803--812, 2011.

Digital Library

[20]

M. Hardt and G. N. Rothblum. A multiplicative weights mechanism for privacy-preserving data analysis. In FOCS, pages 61--70, 2010.

Digital Library

[21]

M. Hardt and K. Talwar. On the geometry of differential privacy. In STOC, pages 705--714, 2010.

Digital Library

[22]

S. P. Kasiviswanathan, H. K. Lee, K. Nissim, S. Raskhodnikova, and A. Smith. What can we learn privately? SIAM J. Comput., 40(3):793--826, 2011.

Digital Library

[23]

K. Nissim, S. Raskhodnikova, and A. Smith. Smooth sensitivity and sampling in private data analysis. In Proceedings of the 39th Annual ACM Symposium on Theory of Computing, San Diego, California, USA, June 11-13, 2007, pages 75--84, 2007.

Digital Library

[24]

C. Peikert, abhi shelat, and A. Smith. Lower bounds for collusion-secure fingerprinting. In SODA, pages 472--479, 2003.

Digital Library

[25]

A. Roth and T. Roughgarden. Interactive privacy via the median mechanism. In STOC, pages 765--774, 2010.

Digital Library

[26]

T. Steinke and J. Ullman. Between pure and approximate differential privacy. In TPDP 2015, 2015.

[27]

G. Tardos. Optimal probabilistic fingerprint codes. J. ACM, 55(2), 2008.

Digital Library

[28]

L. G. Valiant. A theory of the learnable. Commun. ACM, 27(11):1134--1142, Nov. 1984.

Digital Library

[29]

L. G. Valiant. Knowledge infusion. In Proceedings, The Twenty-First National Conference on Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference, July 16-20, 2006, Boston, Massachusetts, USA, pages 1546--1551. AAAI Press, 2006.

Digital Library

Cited By

Kalavasis AKarbasi AMoran SVelegkas GKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Statistical indistinguishability of learning algorithmsProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619044(15586-15622)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619044
Dong WYi KGeerts FNgo HSintos S(2023)Universal Private EstimatorsProceedings of the 42nd ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems10.1145/3584372.3588669(195-206)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3584372.3588669
Dong WLuo QYi K(2023)Continual Observation under User-level Differential Privacy2023 IEEE Symposium on Security and Privacy (SP)10.1109/SP46215.2023.10179466(2190-2207)Online publication date: May-2023
https://doi.org/10.1109/SP46215.2023.10179466
Show More Cited By

Index Terms

Simultaneous Private Learning of Multiple Concepts
1. Theory of computation
  1. Design and analysis of algorithms

Recommendations

Bounds on the sample complexity for private learning and private data release

Learning is a task that generalizes many of the analyses that are applied to collections of data, in particular, to collections of sensitive individual information. Hence, it is natural to ask what can be learned while preserving individual privacy. ...
Characterizing the sample complexity of private learners
ITCS '13: Proceedings of the 4th conference on Innovations in Theoretical Computer Science

In 2008, Kasiviswanathan el al. defined private learning as a combination of PAC learning and differential privacy [16]. Informally, a private learner is applied to a collection of labeled individual information and outputs a hypothesis while preserving ...
Differentially Private Learning of Geometric Concepts

We present efficient differentially private algorithms for learning unions of polygons in the plane (which are not necessarily convex). Our algorithms are $(\alpha,\beta)$--probably approximately correct and $(\varepsilon,\delta)$--differentially private ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ITCS '16: Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science

January 2016

422 pages

ISBN:9781450340571

DOI:10.1145/2840728

Program Chair:
Madhu Sudan
Harvard University

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGACT: ACM Special Interest Group on Algorithms and Computation Theory

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 January 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Simons Foundation
Israel Science Foundation
American Society for Engineering Education
National Science Foundation
Google
Ministry of Science and Technology, Israel
Frankel Center for Computer Science
International Business Machines Corporation

Conference

ITCS'16

Sponsor:

SIGACT

ITCS'16: Innovations in Theoretical Computer Science

January 14 - 17, 2016

Massachusetts, Cambridge, USA

Acceptance Rates

ITCS '16 Paper Acceptance Rate 40 of 145 submissions, 28%;

Overall Acceptance Rate 172 of 513 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
343
Total Downloads

Downloads (Last 12 months)40
Downloads (Last 6 weeks)6

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kalavasis AKarbasi AMoran SVelegkas GKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Statistical indistinguishability of learning algorithmsProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619044(15586-15622)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619044
Dong WYi KGeerts FNgo HSintos S(2023)Universal Private EstimatorsProceedings of the 42nd ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems10.1145/3584372.3588669(195-206)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3584372.3588669
Dong WLuo QYi K(2023)Continual Observation under User-level Differential Privacy2023 IEEE Symposium on Security and Privacy (SP)10.1109/SP46215.2023.10179466(2190-2207)Online publication date: May-2023
https://doi.org/10.1109/SP46215.2023.10179466
Luo QWang YYi KYin HStavrou ACremers CShi E(2022)Frequency Estimation in the Shuffle Model with Almost a Single MessageProceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security10.1145/3548606.3560608(2219-2232)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3548606.3560608
Kaplan HMansour YMatias YStemmer U(2022)Differentially Private Learning of Geometric ConceptsSIAM Journal on Computing10.1137/21M140642851:4(952-974)Online publication date: 7-Jul-2022
https://doi.org/10.1137/21M1406428
Cheu AZhilyaev M(2022)Differentially Private Histograms in the Shuffle Model from Fake Users2022 IEEE Symposium on Security and Privacy (SP)10.1109/SP46214.2022.9833614(440-457)Online publication date: May-2022
https://doi.org/10.1109/SP46214.2022.9833614
Balcer VCheu AJoseph MMao JMarx D(2021)Connecting robust shuffle privacy and pan-privacyProceedings of the Thirty-Second Annual ACM-SIAM Symposium on Discrete Algorithms10.5555/3458064.3458206(2384-2403)Online publication date: 10-Jan-2021
https://dl.acm.org/doi/10.5555/3458064.3458206
Bun MLivni RMoran S(2020)An Equivalence Between Private Classification and Online Prediction2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS)10.1109/FOCS46700.2020.00044(389-402)Online publication date: Nov-2020
https://doi.org/10.1109/FOCS46700.2020.00044
Beimel ANissim KStemmer U(2020)Learning Privately with Labeled and Unlabeled ExamplesAlgorithmica10.1007/s00453-020-00753-zOnline publication date: 3-Aug-2020
https://doi.org/10.1007/s00453-020-00753-z
Alon NLivni RMalliaris MMoran SCharikar MCohen E(2019)Private PAC learning implies finite Littlestone dimensionProceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing10.1145/3313276.3316312(852-860)Online publication date: 23-Jun-2019
https://dl.acm.org/doi/10.1145/3313276.3316312
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents