A Learned Approach to Design Compressed Rank/Select Data Structures

We consider the classical problem of representing, in compressed form, an ordered dictionary S of n elements drawn from the integer universe \( [u] = \lbrace 0, \dots , u-1\rbrace \) while supporting the following operations:

Despite their simple definitions, \( \textsf {rank} \) and \( \textsf {select} \) are powerful enough to solve the ubiquitous predecessor search problem [47], which asks for the largest \( y \in S \) smaller than a given element \( x \in [u] \) . Indeed, it suffices to execute \( y=\textsf {select} (\textsf {rank} (x-1)) \) , where we assumed that \( \textsf {select} (0) = -1 \) to denote the absence of a predecessor for x in S.

Another way of looking at these operations is via the indexing of a binary array \( B_S \) of length \( u, \) which is the characteristic bitvector of S over the universe \( [u] \) . This way, \( \textsf {rank} (x) \) counts the number of bits set to 1 in \( B_S[0 \cdots \cdots x] \) , and \( \textsf {select} (i) \) finds the position of the ith bit set to 1 in \( B_S \) . This interpretation allows generalising the above operations to count and locate symbols in non-binary arrays [30, 33, 45], which are frequently at the core of several text mining and indexing problems.

It is therefore unsurprising that \( \textsf {rank} \) and \( \textsf {select} \) have been studied far and wide since the end of the ’80s [35], with tons of important theoretical and practical results, which we review in Section 1.1. Currently, they are the building blocks of many compact data structures [44] used for designing compressed text indexes [20, 33, 45], succinct trees and graphs [42, 56], monotone minimal perfect hashing [7], sparse hash tables [58], and permutations [6]. Consequently, they have countless applications in bioinformatics [17, 38], information retrieval [43], and databases [1], just to mention a few.

In this article, we show that the problem above has a surprising connection with the geometry of a set of points in the Cartesian plane suitably derived from the integers in S. We then build upon some classical results in computational geometry to introduce a novel data-aware compressed storage and indexing scheme for S that deploys linear approximations of the distribution of these points to “learn” a compact encoding of the input data. We call this novel data structure la_vector because its building blocks are Linear Approximations. We prove theoretical bounds on its time and space performance in the worst case, in the case of input distributions with finite mean and variance, and in terms of the kth order entropy of some of its building blocks. We show that the la_vector can be used in conjunction with other compression schemes, thus originating new hybrid data structures that compare favourably with [50]. Finally, we corroborate these theoretical results with a large set of experiments over a variety of real-world datasets and well-established approaches.

Overall, our theoretical and practical achievements are particularly interesting not only for novel space-time tradeoffs, which add themselves to this highly productive algorithmic field active since 1989 [44], but also because, we argue, they introduce a new way of designing compressed rank/select data structures that deploy computational geometry tools to “learn” the distribution of the input data [24]. As such, we foresee that this novel design may offer research opportunities and stimulate new results from which many applications will hopefully benefit.

Let us assume that \( S = \lbrace x_1, x_2, \ldots , x_n\rbrace \) is a sorted sequence of n distinct integers. We begin by mapping each element \( x_i\in S \) to a point \( (i,x_i) \) in the Cartesian plane, for \( i=1,2,\dots ,n \) [2]. It is easy to see that any function f that passes through all the points in this plane can be thought of as an encoding of S, because we can recover \( x_i \) by querying \( f(i) \) . Clearly, f should be fast to be computed and occupy little space.

Here, we aim at implementing f via a sequence of segments. Segments capture certain data patterns naturally. Any run of consecutive and increasing integers, for example, can be encoded by one segment with slope 1. Generalising, any run of increasing integers with a constant gap g can be encoded by one segment with slope g. Slight deviations from these data patterns can still be captured if we allow a segment to make some “errors” in approximating \( x_i \) at position i, provided that we fix these errors by storing some additional information.

This is the main idea behind our proposal. We reduce the problem of compressing S to the one of “learning” the mapping \( {\textsf {select} :\lbrace 1, \dots , n\rbrace \rightarrow S} \) , which is in turn reduced to the problem of approximating the set of points \( \lbrace (i,x_i)\rbrace _{i=1,\ldots , n} \) via a Piecewise Linear Approximation (PLA), that is a sequence of segments such that every point \( (i, x_i) \) is vertically far from one of these segments by an error bound \( \varepsilon \) , to be fixed later. In some sense, the sequence of segments introduces an “information loss” of \( \varepsilon \) on the integers in S. Among all such sequences of segments (i.e., PLAs), we further aim for the most succinct one, namely, the one with the least amount of segments. This is a classical computational geometry problem that admits an \( O(n) \) -time algorithm by O’Rourke [49]. This algorithm processes each point \( (i,x_i) \) left-to-right, hence for \( i=1, \ldots , n \) , while shrinking a convex polygon in the parameter space of slopes-intercepts. Any coordinate \( (\alpha ,\beta) \) inside the polygon represents a line with slope \( \alpha \) and intercept \( \beta \) that approximates with error \( \varepsilon \) the current set of processed points. When the kth point causes the polygon to be empty, a segment \( (\alpha ,\beta) \) is chosen inside the previous polygon and returned, and a new polygon is started from \( (k,x_k) \) .

We represent the jth segment output by the algorithm above as the triple \( s_j = (r_j, \alpha _j, \beta _j) \) , where \( \alpha _j \) is the slope, \( \beta _j \) is the intercept, and \( r_j \) is the abscissa of the point that started the segment. If \( \ell \) is the number of segments forming the PLA, then we set \( r_{\ell +1}=n \) and observe that \( r_{1}=1 \) . The values \( r_j \) s partition the set of positions \( \lbrace 1,2,\dots ,n\rbrace \) into \( \ell \) ranges so, for any integer i between \( r_j \) and \( r_{j+1} \) (non-inclusive), we use the segment \( s_j \) to approximate the value \( x_i \) as follows:But \( f_j(i) \) is an inexact approximation of \( x_i \) bounded by \( \varepsilon \) . Thus, to turn it into a lossless representation, we complement the values returned by \( f_j \) with an array \( C[1 \cdots \cdots n] \) of integers whose modulo is bounded by \( \varepsilon \) . Precisely, each \( C[i] \) represents the small “correction value” \( x_i - \lfloor f_j(i)\rfloor \) , which belongs to the set \( \lbrace -\varepsilon , -\varepsilon +1, \ldots , -1, 0, 1, \ldots , \varepsilon \rbrace \) . If we allocate \( c \ge 2 \) bits for each correction in C, then the PLA is allowed to err by at most \( \varepsilon = 2^{c-1}-1 \) . We also consider the case \( c=0 \) , for which we set \( \varepsilon =0 \) . We ignore the case \( c=1 \) , because one bit is not enough to distinguish corrections in \( \lbrace -1,0,1\rbrace \) .

The vector C completes our encoding, which we name linear approximation vector (la_vector) and illustrate in Figure 1. Recovering the original sequence S is as simple as scanning the segments \( s_j \) of the PLA and writing the value \( \lfloor f_j(i)\rfloor +C[i]=x_i \) to the output, for \( j=1,\dots ,\ell \) and for \( i=r_j,\dots ,r_{j+1}-1 \) . This process, formalised in Algorithm 1, is appealing in practice, because the array C contains tightly packed integers that are accessed sequentially, and the computation of \( f_j \) is fast, because its values are loaded into three registers when \( s_j \) is first accessed. Moreover, there are no data dependencies among the iterations (as it happens, for example, when integers are delta-coded and a prefix sum is needed).

Recovering a single integer \( x_i \) requires first the identification of the segment \( s_j \) that includes the position i, and then the computation of \( \lfloor f_j(i)\rfloor \, +\, C[i] \) . A binary search over the starting positions \( r_j \) of the segments in the PLA would be enough and takes \( O(\log \ell) \) time, but we will aim for something more sophisticated in terms of algorithmic design and engineering to squeeze the most from this novel approach, as commented in the following sections.

For completeness, we observe that the PGM-index [25] might appear similar to this idea, because it uses PLAs and supports predecessor queries. However, the PGM-index does not compress the input keys but only the index, and it is tailored to the external-memory model, as B-trees. Nonetheless, the PGM-index could take advantage of the la_vector to compress the data stored in its leaves.

To answer \( \textsf {select} (i) \) on the la_vector (either on the plain implementation of Theorem 2.2 or the compressed implementation of Theorem 2.3), we build a predecessor structure \( \mathcal {D} \) on the set \( R = \lbrace r_j \mid 1 \le j \le \ell \rbrace \) and proceed in three steps. First, we use \( \mathcal {D} \) to retrieve the segment \( s_j \) in which i falls into via \( j = \textsf {pred}(i) \) . Second, we compute \( f_j(i) \) , i.e., the approximate value of \( x_i \) given by the segment \( s_j \) . Third, we read \( C[i] \) and return the value \( \lfloor f_j(i)\rfloor + C[i] \) as the answer to \( \textsf {select} (i) \) . The last two steps take \( O(1) \) time. Treating \( \mathcal {D} \) as a black box yields the following result:

If \( \mathcal {D} \) is represented as the characteristic bitvector of the set R augmented with a data structure supporting constant-time predecessor queries (or rank queries, as termed in the case of bitvectors [44]), then we achieve constant-time select by using only \( n+o(n) \) additional bits, i.e., about one bit per integer of S more than what Theorem 2.2 requires. Note that this bitvector encodes R, so the \( \ell \log n \) bits required in Theorem 2.2 for the representation of the \( r_j \) s can be dropped.

Let us compare the space occupancy achieved by the compressed la_vector of Corollary 3.2 to the one of Elias-Fano, namely, \( n (\log \tfrac{u}{n} + 2) + o(n) \) bits [44, Section 4.4], as both solutions support constant-time select. The inequality turns out to bewhere \( d = n/u \) denotes the density of 1s in \( B_S \) .

To solve \( \textsf {rank} (x) \) , it would be sufficient to perform a binary search on the interval \( [1,n] \) to find the largest i such that \( \textsf {select} (i) \le x \) . This naïve implementation takes \( O(t \log n) \) time, because of the implementation of select in \( O(t) \) time by Lemma 3.1.

We can easily improve this solution to \( O(\log \ell + \log n) \) time as follows: First, we binary search on the set of \( \ell \) segments to find the segment \( s_j \) that contains x or its predecessor. Formally, we binary search on the interval \( [1, \ell ] \) to find the largest j such that \( \textsf {select} (r_j) = \lfloor f_j(r_j)\rfloor +C[r_j] \le x \) . Second, we binary search on the \( r_{j+1}-r_j \le n \) integers compressed by segment \( s_j \) to find the largest i such that \( \lfloor f_j(i)\rfloor +C[i] \le x \) . Finally, we return i as the answer to \( \textsf {rank} (x) \) .

Surprisingly, we can further speed up rank queries without adding any redundancy on top of the encoding of Theorem 2.2. The key idea is to narrow down the second binary search to a subset of the elements covered by \( s_j \) (i.e., a subset of the ones in positions \( [r_j, r_{j+1}-1] \) ), which is determined by exploiting the fact that \( s_j \) approximates all these elements by up to an additive term \( \varepsilon \) . Technically, we know that \( |f_j(i) - x_i| \le \varepsilon \) , and we aim to find i such that \( x_i \le x \lt x_{i+1} \) . Hence, we can narrow the range to those \( i\in [r_j, r_{j+1}-1] \) such that \( f_j(i) - \varepsilon \le x \lt f_j(i+1)+\varepsilon \) . By expanding \( f_j(i) = (i-r_j) \cdot \alpha _j + \beta _j \) and noting that f is linear and increasing, we get all candidate i as the ones satisfyingBy solving for i, we getSince i is an integer, we can round the left and the right side of the last inequality, and then we set \( {\mathit {pos} = \lfloor (x - \beta _j) / \alpha _j \rfloor + r_j} \) and \( \mathit {err} = \lceil \varepsilon / \alpha _j \rceil \) , so the searched position i falls in \( [\mathit {pos}-\mathit {err}, \mathit {pos}+\mathit {err}] \) .

The pseudocode of Algorithm 2 exploits these ideas to perform a binary search on the first integers compressed by the segments (Line 1), to compute the approximate rank and the corresponding approximation error (Lines 2 and 3), and finally to binary search on the restricted range specified above (Lines 4–6). As a final note, we observe that \( \alpha _j \ge 1 \) for every j, because the elements in S are increasing, and thus the segments have a slope of at least 1. Consequently, \( \varepsilon /\alpha _j \le \varepsilon \) and the range on which we perform the second binary search has size \( 2\varepsilon \lt 2^c \) , thus this second binary search takes \( O(\log \tfrac{\varepsilon }{\alpha _j})=O(c) \) time.

Note that Lemma 3.3 applies to: (i) the plain la_vector representation provided in Theorem 2.2 (ii) the compressed la_vector representation provided in Theorem 2.3 (the one that compresses \( \beta _j \) s and \( r_j \) s), (iii) the representation provided in Lemma 3.1 (the one supporting select in parametric time t), and (iv) the representation provided in Corollary 3.2 (the one supporting select in constant time).

We can improve the bound of Lemma 3.3 by replacing the binary search at Line 1 of Algorithm 2 with the following predecessor data structure:

Let \( T = \lbrace \textsf {select} (r_j) \mid 1 \le j \le \ell \rbrace \) be the subset of S containing the first integer covered by each segment. We sample one element of T out of \( \Theta (2^c) \) and insert the samples into the predecessor data structure of Lemma 3.4 so \( s = q = \ell /2^c \) and thus \( a = \log \log u \) . Then, we replace Line 1 of Algorithm 2 with a predecessor search followed by an \( O(c) \) -time binary search in between two samples.

We can restrict our attention to the first two branches of the min-formula describing the \( {\rm\small PT} \) term in Lemma 3.4, as the latter two are instead relevant for universe sizes that are super-polynomial in q, i.e., \( \log u = \omega (\log q) \) . The time complexity of rank in Corollary 3.5 then becomes \( O(\min \lbrace \log _w \tfrac{\ell }{2^c}, \log \log \tfrac{u}{\ell }\rbrace + c) \) , where \( w=\Omega (\log u) \) is the word size of the machine.

For input sequences drawn from a distribution with finite mean and variance there exist bounds on the number of segments \( \ell \) , as stated in the following theorem adapted from [19].

Plugging this result into the constant-time select of Corollary 3.2 and the rank implementation of Lemma 3.3, we obtain the following result.

We stress the fact that the data structure of Theorem 4.2 is deterministic. In fact, the randomness is over the gaps between consecutive integers of the input data, and the result holds for any probability distribution as long as the mean and variance are finite. Moreover, according to the experiments in [19], the hypotheses of Theorem 4.1 are very realistic in several applicative scenarios.

Having said that, we observe that the hypothesis “ \( \varepsilon \) is sufficiently larger than \( \sigma \) ” implies that the ratio \( \sigma / \varepsilon \) is much smaller than 1. Hence, it is reasonable to assume that the space bound in Theorem 4.2 is dominated by the term \( n(c + 1) \) which is independent of the universe size while still ensuring constant time select and fast rank operations. If we compare the factor \( c+1 \) present in the space bound of the la_vector with the factor \( \log \tfrac{u}{n} \) present in the space bound of Elias-Fano, we notice that the latter gets larger as the data is sparse ( \( n \ll u \) ). On the other hand, the time complexity of select is constant in both cases, whereas our rank is faster whenever \( \log (n\sigma ^2) \) is asymptotically smaller than \( \log \tfrac{u}{n} \) , which is indeed for \( u = \omega (n^2\sigma ^2) \) .³

In general terms, some results of the previous sections, such as Corollary 3.2 and Lemma 3.3, showed that our la_vector is better than Elias-Fano whenever \( \ell = O(n/\log n) \) . Since Theorem 4.1 proves that \( \ell =\Theta (n/\varepsilon ^2) \) for a large class of input sequences, we can derive that for such sequences our solution is better than Elias-Fano if \( \varepsilon = \omega (\sqrt {\log n}) \) .

So far, we assumed a fixed number of bits \( c \ge 0 \) for each of the n corrections in the la_vector, which is equivalent to saying that the \( \ell \) segments in the PLA guarantee the same error \( \varepsilon = \max (0, 2^{c-1}-1) \) over all the integers in the input set S. However, the input data may exhibit a variety of regularities that allow to compress it further if we use a different c for different partitions of S. The idea of partitioning data to improve its compression has been studied in the past [9, 23, 50, 59, 62], and it will be further developed in this section with regard to our piecewise linear approximations.

We reduce the problem of minimising the space of our rank/select dictionary to a single-source shortest path problem over a properly defined weighted Directed Acyclic Graph (DAG) \( \mathcal {G} \) defined as follows: The graph has n vertices, one for each element in S, plus one sink vertex denoting the end of the sequence. An edge \( (i,j) \) of weight \( w(i,j,c) \) indicates that there exists a segment compressing the integers \( x_i, x_{i+1}, \dots ,x_{j-1} \) of S by using \( w(i,j,c) = (j-i) \, c + \kappa \) bits of space, where c is the bit-size of the corrections, and \( \kappa \) is the space taken by the segment representation in bits (e.g., using the plain encoding of Theorem 2.2 or the compressed encoding of Theorem 2.3). We consider all the possible values of c except \( c=1 \) , because one bit is not enough to distinguish corrections in \( \lbrace -1,0,1\rbrace \) . Namely, we consider \( c \in \lbrace 0, 2, 3, \dots , c_\mathit {max}\rbrace \) , where \( c_\mathit {max}=O(\log u) \) is defined as the correction size that produces one single segment on S. Since each vertex is the source of at most \( c_\mathit {max} \) edges, one for each possible value of c, the total number of edges in \( \mathcal {G} \) is \( O(n\, c_\mathit {max}) = O(n \log u) \) . It is not difficult to prove the following:

Fact 5.1 provides a solution to the rank/select dictionary problem, which minimises the space occupancy of the approaches stated in Theorems 2.2 and 2.3.

Since \( \mathcal {G} \) is a DAG, the shortest path can be computed in \( O(n \log u) \) time by taking the vertices in topological order and by relaxing their outgoing edges [12, Section 24.2]. However, one cannot approach the construction of \( \mathcal {G} \) in a brute-force manner, because this would take \( O(n^2 \log u) \) time and \( O(n \log u) \) space, as each of the \( O(n \log u) \) edges requires computing a segment in \( O(j-i) = O(n) \) time with the algorithm of O’Rourke [49].

To avoid this prohibitive cost, we propose an algorithm that computes a solution on the fly by working on a properly defined graph \( \mathcal {G}^{\prime } \) derived from \( \mathcal {G} \) , taking \( O(n \log u) \) time and \( O(n) \) space. This reduction in both time and space complexity is crucial to make the approach feasible in practice. Moreover, we will see that the obtained solution is not “too far” from the one given by the shortest path in \( \mathcal {G} \) .

Consider an edge \( (i,j) \) of weight \( w(i,j,c) \) in \( \mathcal {G} \) , which corresponds to a segment compressing the integers \( x_i, x_{i+1}, \dots ,x_{j-1} \) of S by using \( w(i,j,c) \) bits of space. Clearly, the same segment compresses any subsequence \( x_a,x_{a+1} \dots , x_{b-1} \) of \( x_i, \dots ,x_{j-1} \) still using c bits per correction. Therefore, the edge \( (i,j) \) “induces” sub-edges of the kind \( (a,b) \) , where \( i \le a \lt b \le j \) , of weight \( w(a,b,c) \) . We observe that the edge \( (a,b) \) may not be an edge of \( \mathcal {G}, \) because a segment computed from position a with correction size c could end past b, thus including more integers on its right. Nonetheless, this property is crucial to define our graph \( \mathcal {G}^{\prime } \) .

The vertices of \( \mathcal {G}^{\prime } \) are the same as the ones of \( \mathcal {G} \) . For the edges of \( \mathcal {G}^{\prime } \) , we start from the subset of edges of \( \mathcal {G} \) that correspond to the segments in the PLAs built for the input set S for all the values of \( c = 0, 2, 3, \ldots , c_{\it max} \) . We call these the full edges of \( \mathcal {G}^{\prime } \) . Then, for each full edge \( (i,j) \) , we generate the prefix edge \( (i,i+k) \) and the suffix edge \( (i+k,j) \) , for all \( k = 1, \dots , j-i \) . This means that we are “covering” every full edge with all of its possible “splits” in two shorter edges having the same correction c as \( (i,j) \) . The total size of \( \mathcal {G}^{\prime } \) is still \( O(n \log u) \) .

We are now ready to show that the graph \( \mathcal {G}^{\prime } \) has a path whose weight is just an additive term far from the weight of the shortest path in \( \mathcal {G} \) . (Notice that this contrasts with the approaches that obtain a multiplicative approximation factor [23, 50].)

We now describe an algorithm that computes the shortest path in \( \mathcal {G}^{\prime } \) without generating the full graph \( \mathcal {G} \) but expanding \( \mathcal {G}^{\prime } \) incrementally to use \( O(n) \) working space. The algorithm processes the vertices of \( \mathcal {G} \) from left to right, while maintaining the following invariant: For \( i=1, \ldots , n+1 \) , each processed vertex i is covered by one segment for each correction size c, and all these segments form the frontier set J.

We begin from vertex \( i=1 \) and compute the \( c_\mathit {max} \) segments that start from i and have any possible correction size \( c = 0, 2, 3, \dots , c_\mathit {max} \) . We set J as the set of these segments. As in the classic step of the shortest path algorithm for DAGs, we do a relaxation step on all the (full) edges \( (i,j) \) , where j is the set of ending positions of the segments in J, that is, we test whether the shortest path to j found so far can be improved by going through i (initially, the shortest-path estimates are set to \( \infty \) for each vertex) and update such shortest path accordingly [12, Section 24.2]. This completes the first iteration.

At a generic iteration i, we first check whether there is a segment in J that ends at i. If so, then we replace that segment with the longest segment starting at i and using the same correction size, computed as usual using the algorithm of O’Rourke [49]. Afterwards, for each full edge \( (a,b) \) that corresponds to a segment in J, we first relax the set of prefix edges of the kind \( (a,i) \) , then we relax the set of suffix edges of the kind \( (i,b) \) . This is depicted in Figure 5.

Given Theorem 5.2, the PLA computed by our algorithm can be used to design a rank/select dictionary that minimises the space occupancy of the solutions based on the approaches of Theorems 2.2 and 2.3. Section 7 will experiment with this approach.

As recalled in Section 1.1, the literature offers a plethora of compressed rank/select dictionaries. Some take into account the statistical or the combinatorial properties of the input, others exploit the compressibility of clusters of consecutive integers. The compression scheme introduced in this article, however, exploits the “geometric properties” of the input data by accommodating their slight deviations from linear trends with the use of small correction values. The choice of the best compression scheme in terms of space occupancy heavily depends on the characteristics of the input data, and thus it is reasonable to expect the best gains in space if we design hybrid solutions that combine several different approaches [4, 50, 59].

In the following, we combine the ideas of Section 5 with the hybrid rank/select dictionary of [50] and thus design an improved hybrid rank/select dictionary. This uses a two-level scheme in which the lower level stores S, properly partitioned into chunks (as detailed below), and the upper level stores, for each lower-level chunk \( x_i,x_{i+1},\dots ,x_j \) , the integer \( x_i=\textsf {select} (i) \) , the length \( j-i+1 \) , and a pointer to the encoding in the lower level. Therefore, the amount of bits stored in the upper level for each chunk is upper bounded by \( F=\log u+2\log n \) .

Following [50], we assign to a generic chunk \( x_i,x_{i+1},\dots ,x_j \) a cost \( w(i,j) \) given by the sum of F and a cost that depends on the encoding of the elements in that chunk. If \( u^{\prime } = x_j-x_i \) is the universe size of the chunk, and \( n^{\prime } = j - i + 1 \) is the number of elements in the chunk, then the cost of that encoding is the minimum of:

Note that the last cost slightly differs from the one in Theorem 2.2. Similarly to Theorem 2.2, we use \( n^{\prime } c \) bits for the vector of corrections. Differently from Theorem 2.2, we use c bits to encode the intercept instead of \( \log n \) bits, because the intercept value is guaranteed to be at most \( \varepsilon \) far from the value i (which is already stored in the upper level of the two-level structure), and thus it can be encoded by shifting i by an amount stored in \( c = \Theta (\log \varepsilon) \) bits. Also, we encode the slope in a word of w bits. Finally, since we need to keep the value c (which possibly changes for each segment), we use additional \( \log c_\mathit {max} \) bits per segment, where \( c_\mathit {max} = O(\log u) \) is defined as in Section 5 as the minimum correction size that produces one single segment on S.

The overall cost in bits of the two-level structure corresponding to a partition P of S into k chunks with endpoints \( 1=i_0,i_1,\dots ,i_k=n \) is given by \( w(P)=\sum _{h=0}^{k-1} w(i_h,i_{h+1}-1) \) . To solve the problem of finding an optimal partition P that minimises \( w(P) \) , we slightly alter the algorithm of [23, 50] to consider also an encoding via segments. The algorithm of [23, 50] finds in \( O(n \log _{1+\epsilon }\tfrac{1}{\epsilon }) \) time and \( O(n) \) space a partition whose cost is only \( 1+\epsilon \) times larger than the optimal one, for any given \( \epsilon \in (0,1) \) . This is done via a left-to-right scan of S, hence for \( i=1,\dots ,n \) , that keeps \( O(\log _{1+\epsilon }\tfrac{1}{\epsilon }) \) sliding windows that start all from i and are such that the kth window covers a chunk \( [i,j] \) such that either \( w(i,j) \le F(1+\epsilon)^k \lt w(i,j+1) \) or \( j=n \) .

A crucial property used in [50] is that computing \( w(i,j) \) for the first three encoders above (namely, EF, BV, and R) takes constant time. Instead, computing whether there is a segment approximating the integers in a chunk requires \( O(n^{\prime }) \) time. Since we need to compute a segment for each value of \( c\in \lbrace 0,2,3,\dots ,c_\mathit {max}\rbrace \) , computing the \( (1+\epsilon) \) -optimal partition P minimising \( w(P) \) takes \( O(c_\mathit {max}\, n^2 \log _{1+\epsilon } \tfrac{1}{\epsilon }) \) time and \( O(n) \) space in the presence of segments, where \( c_\mathit {max} = O(\log u) \) .

In Section 7.5, we experiment with an approach that uses the algorithm of Section 5 to compute a partition in \( O(c_\mathit {max}\, n \log _{1+\epsilon } \tfrac{1}{\epsilon }) \) time and \( O(n \) + \( c_\mathit {max})=O(n) \) space. It operates by keeping a frontier of \( c_\mathit {max} \) segments that overlap the corresponding window and by updating the frontier when the window moves, as we have seen in Section 5 (see the example in Figure 5).

Our experiments were run on a machine with 40 GB of RAM and an Intel Xeon E5-2407v2 CPU.

We have shined a new light on the classical problem of designing rank/select dictionaries by showing a connection between the input data and the geometry of a set of points in a Cartesian plane suitably derived from them. We have introduced new data structures based on this idea and proved their good theoretical bounds and competitive experimental performance with respect to several well-established approaches.

For future work, we mention the study of a relation between classical compressibility measures, such as entropy, and the measure introduced in Section 2.1 based on the number of segments \( \varepsilon \) -approximating the input data. For what concerns Section 2.2, we argue that the space of la_vector can be further improved by computing segments in such a way that the statistical redundancy of the correction values in C is increased. This could be possibly achieved by jointly optimising the space occupied by the segments and the space occupied by the compressed C, playing on both the segments’ lengths and their correction values. We also mention the use of vectorised instructions [37] to achieve fast compression and fast scanning of the corrections in C. Finally, we suggest an in-depth study, design, and experimentation of hybrid rank/select structures, possibly integrating nonlinear models.

We wish to thank the anonymous reviewers for their valuable comments. We also thank the staff of the Green Data Centre at the University of Pisa for providing us with machines and technical support to execute the numerous experiments that have been presented in this article.