Decision List Compression by Mild Random Restrictions
Article No.: 45, Pages 1 - 17
Abstract
A decision list is an ordered list of rules. Each rule is specified by a term, which is a conjunction of literals, and a value. Given an input, the output of a decision list is the value corresponding to the first rule whose term is satisfied by the input. Decision lists generalize both CNFs and DNFs and have been studied both in complexity theory and in learning theory.
The size of a decision list is the number of rules, and its width is the maximal number of variables in a term. We prove that decision lists of small width can always be approximated by decision lists of small size, where we obtain sharp bounds for such approximation. This also resolves a conjecture of Gopalan, Meka, and Reingold (Computational Complexity, 2013) on DNF sparsification.
An ingredient in our proof is a new random restriction lemma, which allows to analyze how DNFs (and more generally, decision lists) simplify if a small fraction of the variables are fixed. This is in contrast to the more commonly used switching lemma, which requires most of the variables to be fixed.
References
[1]
Ryan Alweiss, Shachar Lovett, Kewen Wu, and Jiapeng Zhang. 2020. Improved bounds for the sunflower lemma. In Proccedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing (STOC’20), Konstantin Makarychev, Yury Makarychev, Madhur Tulsiani, Gautam Kamath, and Julia Chuzhoy (Eds.). ACM, 624–630.
[2]
Dana Angluin and Michael Kharitonov. 1995. When won’t membership queries help?J. Comput. Syst. Sci. 50, 2 (1995), 336–355.
[3]
Vikraman Arvind, Johannes Köbler, Sebastian Kuhnert, Gaurav Rattan, and Yadu Vasudev. 2015. On the isomorphism problem for decision trees and decision lists. Theor. Comput. Sci. 590 (2015), 38–54.
[4]
Giulia Bagallo and David Haussler. 1990. Boolean feature discovery in empirical learning. Mach. Learn. 5, 1 (1990), 71–99.
[5]
Paul Beame. 1994. A Switching Lemma Primer. Technical Report. Technical Report UW-CSE-95-07-01, Department of Computer Science.
[6]
Avrim Blum. 1992. Rank-r decision trees are a subclass of r-decision lists. Inform. Process. Lett. 42, 4 (1992), 183–185.
[7]
Aline Bonami. 1970. Étude des coefficients de Fourier des fonctions de. Annal. L’inst. Fourier 20, 2 (1970), 335–402. http://eudml.org/doc/74019.
[8]
Arkadev Chattopadhyay, Meena Mahajan, Nikhil S. Mande, and Nitin Saurabh. 2020. Lower bounds for linear decision lists. (unpublished).
[9]
Andrzej Ehrenfeucht, David Haussler, Michael Kearns, and Leslie Valiant. 1989. A general lower bound on the number of examples needed for learning. Inf. Comput. 82, 3 (1989), 247–261.
[10]
Thomas Eiter, Toshihide Ibaraki, and Kazuhisa Makino. 2002. Decision lists and related boolean functions. Theor. Comput. Sci. 270, 1–2 (2002), 493–524.
[11]
Paul Erdős and Richard Rado. 1960. Intersection theorems for systems of sets. J. Lond. Math. Soc. 35, 1 (1960), 85–90.
[12]
Vitaly Feldman et al. 2007. Efficiency and Computational Limitations of Learning Algorithms. Vol. 68.
[13]
Ehud Friedgut. 1998. Boolean functions with low average sensitivity depend on few coordinates. Combinatorica 18, 1 (1998), 27–35.
[14]
Parikshit Gopalan, Adam Kalai, and Adam R. Klivans. 2008. A query algorithm for agnostically learning DNF? In Proceedings of the 21st Annual Conference on Learning Theory (COLT’08), Rocco A. Servedio and Tong Zhang (Eds.). Omnipress, 515–516. http://colt2008.cs.helsinki.fi/papers/Gopalan-open-question.pdf.
[15]
Parikshit Gopalan, Raghu Meka, and Omer Reingold. 2013. DNF sparsification and a faster deterministic counting algorithm. Comput. Complex. 22, 2 (2013), 275–310.
[16]
Parikshit Gopalan, Raghu Meka, Omer Reingold, Luca Trevisan, and Salil P. Vadhan. 2012. Better pseudorandom generators from milder pseudorandom restrictions. In Proceedings of the 53rd Annual IEEE Symposium on Foundations of Computer Science (FOCS’12). IEEE Computer Society, 120–129.
[17]
David Guijarro, Victor Lavin, and Vijay Raghavan. 2001. Monotone term decision lists. Theor. Comput. Sci. 259, 1–2 (2001), 549–575.
[18]
Thomas Hancock, Tao Jiang, Ming Li, and John Tromp. 1996. Lower bounds on learning decision lists and trees. Inf. Comput. 126, 2 (1996), 114–122.
[19]
Johan Håstad. 1987. Computational Limitations of Small-depth Circuits. MIT Press, Cambridge, MA.
[20]
Jeffrey C. Jackson. 1997. An efficient membership-query algorithm for learning DNF with respect to the uniform distribution. J. Comput. Syst. Sci. 55, 3 (1997), 414–440.
[21]
Michael Kearns, Ming Li, Leonard Pitt, and Leslie Valiant. 1987. On the learnability of boolean formulae. In Proceedings of the 19th Annual ACM Conference on Theory of Computing, Vol. 1987. Citeseer, 285–295.
[22]
Zander Kelley. 2020. An improved derandomization of the switching lemma. Electron. Colloquium Comput. Complex. 27 (2020), 182.
[23]
Adam R. Klivans, Homin K. Lee, and Andrew Wan. 2010. Mansour’s conjecture is true for random DNF Formulas. In Proceedings of the 23rd Conference on Learning Theory (COLT’10), Adam Tauman Kalai and Mehryar Mohri (Eds.). Omnipress, 368–380.
[24]
Adam R. Klivans and Rocco A. Servedio. 2003. Boosting and hard-core set construction. Mach. Learn. 51, 3 (2003), 217–238.
[25]
Ron Kohavi and Scott Benson. 1993. Research note on decision lists. Mach. Learn. 13, 1 (1993), 131–134.
[26]
Matthias Krause. 2006. On the computational power of boolean decision lists. Comput. Complex. 14, 4 (2006), 362–375.
[27]
Nathan Linial, Yishay Mansour, and Noam Nisan. 1993. Constant depth circuits, fourier transform, and learnability. J. ACM 40, 3 (1993), 607–620.
[28]
Shachar Lovett and Jiapeng Zhang. 2019. DNF sparsification beyond sunflowers. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing (STOC’19).454–460.
[29]
Yishay Mansour. 1995. An O(n^(log log n)) learning algorithm for DNT under the uniform distribution. J. Comput. Syst. Sci. 50, 3 (1995), 543–550.
[30]
Ziv Nevo and Ran El-Yaniv. 2002. On online learning of decision lists. J. Mach. Learn. Res. 3, (Oct.2002), 271–301.
[31]
Ryan O’Donnell. 2014. Analysis of Boolean Functions. Cambridge University Press.
[32]
Alexander A. Razborov. 1995. Bounded arithmetic and lower bounds in boolean complexity. In Feasible Mathematics II. Springer, 344–386.
[33]
Alexander A. Razborov. 2015. Pseudorandom generators hard for k-DNF resolution and polynomial calculus resolution. Ann. Math. (2015), 415–472.
[34]
Ronald L. Rivest. 1987. Learning decision lists. Mach. Learn. 2, 3 (1987), 229–246.
[35]
Nathan Segerlind, Sam Buss, and Russell Impagliazzo. 2004. A switching lemma for small restrictions and lower bounds for k-DNF resolution. SIAM J. Comput. 33, 5 (2004), 1171–1200.
[36]
Avishay Tal. 2017. Tight bounds on the fourier spectrum of AC0. In Proceedings of the 32nd Computational Complexity Conference (CCC’17), Ryan O’Donnell (Ed.), LIPIcs, Vol. 79. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 15:1–15:31.
[37]
Luca Trevisan and Tongke Xue. 2013. A derandomized switching lemma and an improved derandomization of AC0. In Proceedings of the 28th Conference on Computational Complexity (CCC’13). IEEE Computer Society, 242–247.
[38]
György Turán and Farrokh Vatan. 1997. Linear decision lists and partitioning algorithms for the construction of neural networks. In Foundations of Computational Mathematics. Springer, 414–423.
[39]
Karsten A. Verbeurgt. 1990. Learning DNF under the uniform distribution in quasi-polynomial time. In Proceedings of the 3rd Annual Workshop on Computational Learning Theory (COLT’90), Mark A. Fulk and John Case (Eds.). Morgan Kaufmann, 314–326. http://dl.acm.org/citation.cfm?id=92659.
[40]
Fulton Wang and Cynthia Rudin. 2015. Falling rule lists. In Artificial Intelligence and Statistics. 1013–1022.
[41]
Ian H. Witten, Eibe Frank, Mark A. Hall, and Christopher J. Pal. 2016. Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann.
Index Terms
- Decision List Compression by Mild Random Restrictions
Recommendations
Decision list compression by mild random restrictions
STOC 2020: Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of ComputingA decision list is an ordered list of rules. Each rule is specified by a term, which is a conjunction of literals, and a value. Given an input, the output of a decision list is the value corresponding to the first rule whose term is satisfied by the ...
From DNF compression to sunflower theorems via regularity
CCC '19: Proceedings of the 34th Computational Complexity ConferenceThe sunflower conjecture is one of the most well-known open problems in combinatorics. It has several applications in theoretical computer science, one of which is DNF compression, due to Gopalan, Meka and Reingold (Computational Complexity, 2013). In ...
DNF sparsification beyond sunflowers
STOC 2019: Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of ComputingThere are two natural complexity measures associated with DNFs: their size, which is the number of clauses; and their width, which is the maximal number of variables in a clause. It is a folklore result that DNFs of small size can be approximated by ...
Comments
Information & Contributors
Information
Published In
December 2021
283 pages
ISSN:0004-5411
EISSN:1557-735X
DOI:10.1145/3484923
- Editor:
- Venkatesan Guruswami
Copyright © 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2021
Accepted: 01 September 2021
Revised: 01 March 2021
Received: 01 March 2020
Published in JACM Volume 68, Issue 6
Check for updates
Author Tags
Qualifiers
- Research-article
- Refereed
Funding Sources
- NSF
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 268Total Downloads
- Downloads (Last 12 months)108
- Downloads (Last 6 weeks)21
Reflects downloads up to 30 Aug 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in