Unlabeled Compression Schemes Exceeding the VC-dimension arXiv:1811.12471v1 [math.CO] 29 Nov 2018 Dömötör Pálvölgyi∗ and Gábor Tardos† December 3, 2018 Abstract In this note we disprove a conjecture of Kuzmin and Warmuth claiming that every family whose VC-dimension is at most d admits an unlabeled compression scheme to a sample of size at most d. We also study the unlabeled compression schemes of the joins of some families and conjecture that these give a larger gap between the VC-dimension and the size of the smallest unlabeled compression scheme for them. 1 Introduction Terminology: if S is a subset of the domain of a function f , then we call the restriction g = f |S the trace of f on S and we also call f an extension of g. Consider a finite set B, and fix a family F of functions B → {0, 1}. For f ∈ F and S ⊆ B we call the trace f |S a partial function of the family F. These are studied extensively in learning theory, where our goal is to reconstruct f |S from some part of it. Definition 1 (Littlestone and Warmuth [3]). A (labeled) compression scheme for F is a pair of operations (α, β) such that • α takes a partial function g of F as an input (called a labeled sample) and returns a trace of g, • β takes the output of α as input and returns an arbitrary function f : B → {0, 1}, • β(α(g)) is an extension of g for any partial function g of F. That is, instead of f |S , it is enough to store α(f |S ) so that we can fully recover the value of f over S. The size of the compression scheme (α, β) is the maximum size of the domain of α(g). We denote by LCS(F) the minimum size of a compression scheme for F. Remark 2. Notice that it is not required to be able to reconstruct S from α(f |S ). Remark 3. β(α(f |S )) is not required to be from F. ∗ MTA-ELTE Lendület Combinatorial Geometry Research Group, Institute of Mathematics, Eötvös Loránd University (ELTE), Budapest, Hungary. Research supported by the Lendület program of the Hungarian Academy of Sciences (MTA), under grant number LP2017-19/2017. † Supported by the Cryptography “Lendület” project of the Hungarian Academy of Sciences and by the National Research, Development and Innovation Office, NKFIH projects K-116769 and SNN-117879. 1 Definition 4 (Vapnik-Chervonenkis [5]). Let F be a family of functions B → {0, 1}. We say that F shatters X ⊆ B if every function g : X → {0, 1} has an extension in F. The VC-dimension of F, VC(F), is defined as the size of the largest X that is shattered by F. Littlestone and Warmuth [3] observed that LCS(F) ≥ VC(F)/5 always holds but could not give any compression scheme for general families whose size depended only on VC(F). Floyd and Warmuth [1] conjectured that LCS(F) ≤ VC(F) always holds. (There are simple examples that show that this would be sharp.) Warmuth [6] even offered $600 reward for a proof that a compression scheme of size O(d) always exists, but this has been proved only in special cases.∗ In 2015, Moran and Yehudayoff [4] have managed to prove that a compression scheme exists whose size depends only on VC(F), but their bound is exponential in VC(F). Definition 5 (Kuzmin and Warmuth [2]). An unlabeled compression scheme for F is a pair of operations (α, β) such that • α takes a partial function g with domain S (called a labeled sample) and returns a α(g) (called the compressed sample), which is a subset of S, • β takes the output of α as input and returns an arbitrary function f : B → {0, 1}, • β(α(g)) is an extension of g for any partial function g of F. That is, unlike in the case of labeled compression schemes, we do not store the value of f on the compressed sample, but only some selected sample points. The size of the unlabeled compression scheme (α, β) is the maximum size of α(g) for any partial function g. We denote by UCS(F) the minimum size of an unlabeled compression scheme for F. Note that UCS(F) ≥ LCS(F) trivially holds. Kuzmin and Warmuth [2] have proved that UCS(F) ≥ VC(F) and conjectured that equality might hold for every family (a strengthening of the earlier conjecture of Floyd and Warmuth).† We disprove this last conjecture in a very weak sense; we exhibit a small family C5 for which VC(C5 ) = 2 but UCS(C5 ) = 3. We also discuss possible ways to amplify this gap, but at the moment we do not know any family F with UCS(F) > VC(F) for which UCS(F) ≥ 4. (Although a computer search could possibly find such a family - we exhibit some likely candidates.) Lower bound for C5 2 Here we define the family C5 for which UCS(C5 ) = 3 > VC(C5 ) = 2, and prove these equalities. The base set of C5 is five elements and |C5 | = 10; see Figure 1. We think of the base set B of C5 as the vertices of a regular pentagon. A 0-1 function on this base set belongs to C5 if and only if it takes the values 1-0-0-1 on some four consecutive vertices. As we have later found out, this is known in the learning theory literature as ‘Warmuth’s example.’ He constructed it as a simple example of a containment maximal family with VC(C5 ) = 2 that does not reachP the maximal size of such a family given by the Sauer-Shelah lemma, which in
this case would be 2i=0 5i = 16.
P Floyd and Warmuth [1] claimed to have proved it for families of VC-dimension d whose size is di=0 ni , i.e., the maximum size allowed by the Sauer-Shelah lemma, but recently an error was discovered in their argument. † Similarly to the labeled case, they also made a claim about maximum size families, which seems to contain the same error. ∗ 2 1 0 1 1 0 1 0 1 0 0 Figure 1: C5 consists of the 5 rotations of the above sets. We will use the property that for any subset S ⊂ B of size 3 there are 7 possibilities for the trace f |S for f ∈ C5 . If S consists of three consecutive vertices, then f |S cannot be constant 0, while if S consists of three non-consecutive vertices the constant 1 trace is not possible. Note that this implies that C5 shatters no three element set but it shatters all two element sets, so its VC-dimension is 2. We identify the base set B of C5 with the residue classes modulo 5, with the neighbors of the vertex i ∈ B being i + 1 and i − 1. Theorem 6. UCS(C5 ) = 3. Proof. It is easy to construct an unlabeled compression scheme of size 3: α can keep the sample points where the value of the function is 1, and the reconstruction function β returns 1 at every place contained in the compressed sample, and 0 everywhere else. Thus, we only need to prove that UCS(C5 ) ≥ 3. Suppose by contradiction that there is an unlabeled compression scheme (α, β) of size two. Let X be a size 3 subset of the domain. As we noted above, there are exactly 7 partial functions g : X → {0, 1} of C5 . Clearly, α(g) must be a distinct proper subset of X for each. As there are 7 such subsets, we must have a 1-1 correspondence here. In particular, for all Y ( X, the β(Y )|X must be distinct partial functions of C5 . Let J be the set of three consecutive positions in the domain and i ∈ J. Let g be the constant 0 partial function defined on J \ {i} and Y = α(g). Here β(Y )|J is a partial function of C5 extending g, so it must be 0 on J \ {i} and 1 on i. Now β({i})|J must be another partial function of C5 , therefore β({i})|(J\{i}) cannot be constant 0. A symmetric argument shows that if K is the set of three non-consecutive positions and i ∈ K, then β({i})|(K\{i}) is not constant 1. The observations above imply that β({i})(i − 1) = 1. Indeed, if β({i})(i − 1) = 0, then applying the observation in the previous paragraph for J = {i − 2, i − 1, i} we obtain β({i})(i − 2) = 1 and considering J = {i − 1, i, i + 1} we obtain β({i})(i + 1) = 1, but this contradicts our observation about K = {i − 2, i, i + 1}. A similar argument shows β({i})(i + 1) = 1 as well as β({i})(i − 2) = β({i})(i + 2) = 0. The only remaining value, namely β({i})(i) therefore completely determines β({i}). Suppose β({i})(i) = 1 holds for at least three different values of i; then it must hold for two consecutive values, say i and i + 1. This completely determines β({i}) and β({i + 1}) and these functions coincide on X = {i − 2, i, i + 1} contradicting our observation that for distinct proper subsets Y of X, the β(Y )|X must also be distinct. 3 Alternatively we must have β({i})(i) = 0 for at least three different values of i. Then it also holds for two non-consecutive values, say i − 1 and i + 1. This completely determines β({i − 1}) and β({i + 1}) and these functions coincide on X = {i − 1, i, i + 1}, a contradiction again. The contradictions prove the theorem. Remark 7. Note that in the above proof we have in fact showed that there is no compression scheme already in the case when the sample consists of at most 3 values. 3 Upper bounds for C5 ’s In this section we sketch some upper bounds, i.e., give unlabeled compression schemes for certain families. When we receive a sample f |S , we interpret it as receiving a collection of 0’s and 1’s, and we interpret the compression as keeping some of them (though we only keep the locations, not the values). In the case of C5 , when we receive a sample that contains 3 identical values, then we call them a triple 0 or a triple 1, depending on the value. Recall that a triple 1 can only occur at 3 consecutive positions, and a triple 0 can only occur at 3 non-consecutive positions, so the set of positions determines whether it is a triple 0 or a triple 1. Definition 8. The join of two families of functions F ∗ G = {(f, g) | f ∈ F, g ∈ G} is a family over the disjoiont union of there base sets where (f, g)(x) = f (x) if x belongs to the base set of F and g(x) if x belongs to the base set of G. When we take the join of several copies of the same family, we use the notation F ∗n = F . . ∗ F} . | ∗ .{z n times We obviously have VC(F ∗G) = VC(F)+VC(G), but for compression schemes only UCS(F ∗G) ≤ UCS(F) + UCS(G) follows from the definition, and equality does not always hold, as the following statement shows. Recall that UCS(C5 ) = 3 by Theorem 6. Proposition 9. UCS(C5 ∗ C5 ) ≤ 5. Sample no triples (1) triple 1 in C5 (1) triple 0 in C5 (2) triple 1 in C5 (2) triple 0 in C5 keep keep keep keep Compression keep all 1’s triple and 1’s in triple and 0’s in triple and 0’s in triple and 1’s in (2) C5 (2) C5 (1) C5 (1) C5 Decoding kept to 1, rest 0 triple from position, (2) kept in C5 same triple from position, (1) kept in C5 opposite Table 1: Compressing C5 ∗ C5 . Proof. For the proof we need to give an unlabeled compression scheme (α, β). There are several (1) (2) possible schemes, one is sketched in Table 1. We write C5 and C5 for the base sets of the two copies of C5 . The compression α depends on whether there are, and what type of triples in the labeled sample restricted to the base sets of the two copies of C5 . We denote these base sets by (1) (2) C5 and C5 . If neither of them contains a triple, we just keep the 1’s in the labeled sample. (1) (2) If C5 contains a triple 1, but C5 does not contain a triple 1, then we still just keep the 1’s. 4 (1) (2) If C5 contains a triple 0, but C5 does not contain a triple 0, then we keep all the 0’s in the labeled sample. (2) (1) (2) If C5 contains a triple 1, but C5 does not contain a triple 0, then keep the triple 1 from C5 , (1) and the 0’s from C5 . (2) (1) (2) If C5 contains a triple 0, but C5 does not contain a triple 1, then keep the triple 0 from C5 , (1) and the 1’s from C5 . (1) (2) Note that if the compressed sample contains three positions from either C5 or C5 , then those positions formed a triple in the labeled sample and it was a triple 1 in case of three consecutive positions and a triple 0 in case of three non-consecutive positions. This means that the compressed sample determines which one of the five rules was used to obtain it and the decoding β can be constructed accordingly. Finally, notice that exactly one of the 5 above cases happens for every sample. (Although note that for us it would be sufficient if at least one of them happened for every sample.) This raises the question of how UCS(F ∗n ) behaves when n → ∞. We can prove neither any lower bound that would be better than n·VC(F) for any F at all (notice that Proposition 9 only provides an upper bound, but we do not know whether in general UCS(F ∗ G) ≥ UCS(F) + UCS(G) − 1 holds or not), nor show that UCS(F ∗n ) ≤ (1 + o(1))n · VC(F) for every F. We make the following conjecture. Conjecture 10. limn→∞ UCS(C5∗n ) n exists and is strictly larger than 2. We can prove that UCS(C5∗n ) ≤ 2n + 1 for n ≤ 5. Since the compression schemes are based on similar ideas, we only sketch the scheme for n = 5. Proposition 11. UCS(C5∗5 ) ≤ 11. Sample Compression keep all 1’s (i) keep triple 1 in C5 (j) and 0’s in other C5 ’s keep 0’s fix two triple 0’s and one triple 1; keep non-central triples and central element of central triple, and 1’s from rest no triple 1 (i) triple 1 in some C5 but no triple 0 anywhere exactly one triple 0 exactly one triple 1 and least two triple 0’s least two triple 1’s and least two triple 0’s, and fifth does not have exactly one 1 two triple 1’s and least two triple 0’s, and fifth has exactly one 1 keep triple 1’s and central elements of triple 0’s, and 1’s from fifth keep triple 1’s and non-central elements of triple 0’s, and 1 from fifth Table 2: Compressing C5∗5 . (0) (4) Proof. We denote the 5 copies of C5 ’s by C5 , . . . , C5 , with indexing mod 5. 5 (i) Among any three positions in a single C5 there is a unique “central” element: the one that is equidistant from the other two elements. We use that the two non-central elements determine the central element uniquely. Although the central element is not enough to determine the other two elements, it becomes enough once we know whether they are the positions in a triple 0 or a triple 1. (i) (j) (k) Similarly, among any three distinct sets C5 , C5 and C5 , there is a unique central one, whose (0) (2) (3) index is equidistant (modulo 5) from the other two indices. E.g., from C5 , C5 and C5 the (0) (0) (3) (4) (4) central one is C5 , while from C5 , C5 and C5 the central one is C5 . We use again that the non-central copies determine the central one uniquely. The compression algorithm is sketched in Table 2. This Table needs to be interpreted in a similar fashion as Table 1, this time we omit the lengthy description of the case analysis. Note that for some labeled samples there are more rules to choose from for the compression – in this case, we pick arbitrarily. It is important, however that there is always at least one rule that applies. We have also omitted the decompression rules, as the compressed sample always determines which rule was used to obtain it. To prove this statement, notice that we only keep three position (i) of the same C5 if they form a triple in the labeled sample. If the first rule is used, no triple is kept. In case the second or third rule is used, a single triple 1 or triple 0 is kept, respectively. If the fourth rule is used, then two triples are kept, not both triple 1’s. Finally if either of the last two rules are used, then at least two triple 1’s are kept. The compressed sample produced by the last two rules (i) are distinguished by the number of elements kept in the sets C5 : if it is 3 + 3 + 2 + 2 + 1 in some order, then the last rule was used, otherwise the fifth rule. Once we know which rule produced the compressed sample the decoding can be done accordingly. 4 Further results In this section we mention some further results. We start by defining some further families. C5− is obtained from C5 by deleting one function. Because of the symmetry, it does not matter which one, so we delete the function 0-1-1-1-0. Here we represent functions by the sequence of their values on 0, 1, 2, 3, 4. In this family, still any two positions can take any values (4 possibilities each), but for some triples we have only 6 possibilities (instead of 7). C4 is the restriction of C5 to four elements of the base set. Again, by symmetry it does not matter which four, so we delete the central element 2. This is useful, because this way C4 also becomes a restriction of C5− . Proposition 12. UCS(C4 ) = UCS(C5− ) = 2. Proof. The lower bounds follow from 2 = VC(C4 ) ≤ UCS(C4 ) ≤ UCS(C5− ). For the upper bound, we need to give a compression scheme of size two for C5− . A possible algorithm is sketched in Table 3. Here we list the decoding of compressed samples only. We maintain a symmetry for the reflection to the central element: If the compressed sample B is obtained from another compressed sample A by reflection, then the decoding β(B) is also obtained from β(A) the same way. Accordingly, we only list one of A and B in the Table. We omit the lengthy case analysis of why this compression scheme works. Now we continue by definining two more families. 6 Compression ∅ x-.-.-.-. .-x-.-.-. .-.-x-.-. x-x-.-.-. x-.-x-.-. x-.-.-x-. x-.-.-.-x .-x-x-.-. .-x-.-x-. Decoding 1-0-0-0-1 0-0-1-0-1 1-1-0-0-1 1-0-1-0-1 0-1-0-0-1 0-1-0-0-1 0-0-1-1-1 0-1-0-1-0 1-1-1-0-0 0-1-0-1-0 Table 3: Compressing C5− ; elements of the compressed sample are marked with an x. P (k) is the family of all 2k boolean functions on a base set of k elements. Notice that P (k) = P (1)∗k . As P (k) shatters its entire base set, we have VC(P (k)) = k. We also have UCS(P (k)) = k as VC(P (k)) ≤ UCS(P (k)) and UCS(P (k) ≤ k is shown by the simple unlabeled compression scheme that keeps the 1’s in the labeled sample. On the other hand, LCS(P (k)) can be smaller, e.g., LCS(P (2)) = 1. W6 is a symmetrizing extension of C5 , with the same number of functions, but one more base element. One can obtain it from C5 by adding an extra element to the base and extending each function in the family to the new element such that the function has three zeros and three ones. Figure 2 depict two functions of W6 . The other eight functions are the rotations of these two. In the family W6 the extra element plays no special role, in fact, W6 is two-transitive, i.e., any pair of elements of its base set can be mapped to any other pair of elements with an automorphism. If we convert the functions of W6 to 3-element sets, we get the unique 2 − (6, 3, 2) design. Since W6 is an extension of C5 , VC(C5 ) ≤ VC(W6 ) and UCS(C5 ) ≤ UCS(W6 ) – it is easy to check that we have equality in both cases, i.e., VC(W6 ) = 2 and UCS(W6 ) = 3. 1 0 1 1 1 1 0 0 1 0 0 0 Figure 2: W6 consists of the 5 rotations of the above sets. Some further non-trivial upper bounds can be obtained for the joins involving these families. Proposition 13. UCS(W6 ∗ P (1)) = 3. 7 Sample extra is not 1 extra is 1 and triple 0 extra is 1, no triple 0 Compression keep 1’s of W6 keep triple 0 keep extra and 0’s Decoding kept 1, others 0 kept 0, others 1 extra 1, rest of kept 0, others 1 Table 4: Compressing W6 ∗ P (1). Proof. The compression algorithm is sketched in Table 4, with ‘extra’ denoting the only bit of the base set of P (1). Note that C5 ∗ P (1) is obtained from W6 ∗ P (1) by restricting the base set and such a restriction cannot increase the value of UCS, so this also implies UCS(C5 ∗ P (1)) = 3. From this we can easily get another proof for UCS(C5 ∗ C5 ) ≤ 5 as follows. We have C5 ⊂ P (1) ∗ C4 , thus UCS(C5 ∗ C5 ) ≤ UCS(C5 ∗ P (1) ∗ C4 ) ≤ UCS(C5 ∗ P (1)) + UCS(C4 ) ≤ 3 + 2, using Proposition 12. Proposition 14. UCS(W6 ∗ W6 ) ≤ 5. Proof. This compression goes similarly to the one presented in Table 1 for C5 ∗C5 . In fact, we can use exactly the same compression scheme unless we get two triples in both W6 ’s, i.e., a labeled sample that contains all 12 elements of the base. There are 10 · 10 = 100 possibilities for such a sample, and for each we can pick a compression that keeps
at least at

4 elements

from

least
one
of
the
two
copies of W6 , as these were not used yet. There are 65 · 60 + 64 · 61 + 64 · 60 + 60 · 64 + 61 · 64 + 60 · 65 = 222 such possible compressed samples, we can use a distinct one for each of the 100 problematic labeled samples. This makes the decoding possible. We end by a summary of the most important questions left open. Summary of main open questions • Is UCS(F) − VC(F) bounded? • Is UCS(F ∗ G) ≥ UCS(F) + UCS(G) − 1? • How does UCS(C5∗n ) behave? Does lim UCS(n ∗ F)/n exist? • Is there a k for every F such that UCS(F ∗ P (k)) = VC(F) + k? Remarks and acknowledgment We would like to thank Tamás Mészáros, Shay Moran and Manfred Warmuth for useful discussions and calling our attention to new developments. References [1] S. Floyd and M. K. Warmuth, Sample compression, learnability, and the Vapnik-Chervonenkis dimension, in Machine Learning, 21(3):269–304, 1995. [2] D. Kuzmin and M. K. Warmuth, Unlabeled Compression Schemes for Maximum Classes, in Proceedings of the 18th Annual Conference on Computational Learning Theory (COLT 05), Bertinoro, Italy, pp. 591–605, June 2005. 8 [3] N. Littlestone and M. K. Warmuth, Relating data compression and learnability. Unpublished manuscript, obtainable at http://www.cse.ucsc.edu/~manfred, June 10 1986. [4] S. Moran and A. Yehudayoff, Sample compression schemes for VC classes, to appear in the Journal of the ACM. [5] V. N. Vapnik, A. Ya. Chervonenkis, On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities, Theory of Probability & Its Applications. 16(2):264-280, 1971. [6] M. K. Warmuth, Compressing to VC dimension many points, in Proceedings of the 16th Annual Conference on Learning Theory (COLT 03), Washington D.C., USA, August 2003. Springer. Open problem. https://users. soe.ucsc.edu/~manfred/pubs/open/P1.pdf. 9