Better Locally Private Sparse Estimation Given Multiple Samples Per User

We introduce necessary notations. For any vector $x$, let $x^{i}$ denote the $i$-th element of $x$. Let $x^{\{i_{1},\cdots,i_{j}\}}$ be a slicing vector of $x$, whose $j$-th elements is $x^{i_{j}}$. Let $\|x\|_{p}$ be the $\ell_{p}$ norm of $x$ for $0\leq p\leq\infty$. We will evaluate the estimation error by the squared loss, i.e. $\|\widehat{\beta}-\beta^{*}\|_{2}^{2}$. For matrix $A$, let $\lambda_{i}(A)$ denote the $i$-th largest singular value of $A$. Throughout this paper, we use the notation $a_{n}\lesssim b_{n}$ and $a_{n}\gtrsim b_{n}$ to denote that there exist positive constant $c$ and $c^{\prime}$ such that $a_{n}\leq cb_{n}$ and $a_{n}\geq c^{\prime}b_{n}$, for all $n\in\mathbb{N}$. We use $a=\mathcal{O}(b)$ if $a\lesssim b$. We denote $a_{n}\asymp b_{n}$ if $a_{n}\lesssim b_{n}$ and $b_{n}\lesssim a_{n}$. Let $a\vee b=\max(a,b)$ and $a\wedge b=\min(a,b)$. Besides, for any set $A\subset\mathbb{R}^{d}$, the diameter of $A$ is defined by $\mathrm{diam}(A):=\sup_{x,x^{\prime}\in A}\|x-x^{\prime}\|_{2}$.

Suppose we have $n$ users. The $i$-th user has $m$ i.i.d. samples $(X_{i},y_{i})=\{(X_{i,j},y_{i,j}),j=1,\cdots,m\}$ from distribution $\mathrm{P}$ on domain $\mathcal{X}\times\mathcal{Y}\subseteq\mathbb{R}^{d}\times\mathbb{R}$. We consider the classical sparse linear regression. Let each $X_{i,j}$ be i.i.d. sub-Gaussian. Moreover, $\Sigma=\mathbb{E}[XX^{\top}]$ denote the covariance matrix of the marginal distribution. Assume $C_{X}^{-1}\leq\lambda_{d}(\Sigma)\leq\lambda_{1}(\Sigma)\leq C_{X}$ for some constant $C_{X}>1$. For mean zero sub-Gaussian random variable $\sigma$, conditional distribution $\mathrm{P}_{Y|X}$ and its coefficients $\beta^{*}$ are described by

which has $s^{*}$-sparsity and non-zero entries bounded away from 0. Without loss of generality, we assume the first $s^{*}$ elements of $\beta^{*}$ are non-zero.

We adopt the following setting for privacy constraints. Any estimation of $\beta^{*}$ is considered as a random variable, while its construction process with respect to the data is user-level locally differentially private (ULDP). We consider the sequential interactive case where the private observation $U_{i}$ is decided only by its local samples $(X_{i},y_{i})$ and previous observations $U_{1},\cdots,U_{i-1}$. The rigorous definition of (pure) ULDP is as follows.

ULDP reduces to the conventional item-level LDP for $m=1$. Besides being more practically reasonable (Cummings et al., 2022), ULDP is also a more stringent definition than item-level LDP. To achieve $\varepsilon$-ULDP by trivially using group privacy, each item must use a significantly smaller budget $\varepsilon/m$. Conversely, on the curator side, inference of any single item is no easier than inference of the whole user, which means each item is as safe as $\varepsilon$-LDP against the curator. As a compromise, each item should expose information to its group mates. The requirement is typically acceptable, such as when each user has multiple records on a personal cellphone, or when data sources can be clustered into small groups where secrete information is safely shared.

Extensive studies have been conducted focusing on the central DP setting for linear regression model in low dimensions (Wang, 2018; Avella-Medina et al., 2023; Alabi et al., 2022; Arora et al., 2022; Amin et al., 2023) and high dimensions (Kifer et al., 2012; Talwar et al., 2015; Kumar & Deisenroth, 2019; Zhang & Zhang, 2021; Cai et al., 2021; Hu et al., 2022; Khanna et al., 2023a, b; Raff et al., 2023). Despite variations in settings and assumptions, state-of-the-art results (Liu et al., 2022; Varshney et al., 2022; Cai et al., 2023) indicate a general error rate of $\mathcal{O}(s^{*}\log d/(n\varepsilon^{2}))$ for squared loss, where the dependency on the dimension of feature space is $\log d$. Thus, $d$ can be exponentially large in $n\varepsilon^{2}$ to ensure consistent estimation.

This is not the case in local setting, of which there is still a lack of understanding compared to the central one. Several works addressed the problem focusing on the optimization error (Smith et al., 2017; Zheng et al., 2017). Both works assumed $\mathrm{diam}(\mathcal{X})\leq 1$ and do not generalize to many practical settings, such as when all features are i.i.d. and therefore $\mathrm{diam}(\mathcal{X})=\mathcal{O}(\sqrt{d})$. As for statistical estimation, Duchi et al. (2018) showed the matching upper and lower bounds for low dimensional, non-interactive linear regression are $\mathcal{O}(d/(n\varepsilon^{2}))$. Wang & Xu (2019) first provided the lower bound $\mathcal{O}(d/(n\varepsilon^{2}))$ for LDP linear regression with 1-sparsity, which is then generalized to $s$-sparsity by Zhu et al. (2023). In summary, these prohibitive results indicate that there exists no meaningful approach when $d\asymp n\varepsilon^{2}$, which is often the case in practice.

Our approach utilize a selection-estimation strategy, which is shown to be advantages under many situation. Under non-private setting, Wang et al. (2011); Liang et al. (2023) select candidate variables by aggregating Lasso fitted on random subsamples, which is also adopted with privacy (Kifer et al., 2012). More recently, the strategy has been used for communication-constrained learning (Duchi & Rogers, 2019; Barik & Honorio, 2020; Acharya et al., 2021). Note that our method is also communication efficient, as each user sends only $1$ bit of information. Acharya et al. (2021) tackled LDP sparse discrete distribution estimation by selecting the support variables. However, this is only feasible for such specific problems where users can provide useful information about which variables are potentially non-zero given only one sample. Their result does not generalize to other problems.

During the preparation of the camera-ready version of this paper, Kent et al. (2024) appeared online and analyzed sparse mean estimation under ULDP. We share some results with their conclusions, including the negative results for $m\leq s^{*}\log d$, the established rates, and a support estimation type estimator. However, we primarily consider the case where $s^{*},\log d\lesssim n\varepsilon^{2},m\lesssim d$, whereas their analysis is more comprehensive, considering other regions and, more importantly, identifying the phase transition.

We introduce the related minimax results of locally private sparse linear regression. For any loss function $\ell$ (squared loss in our case), the minimax convergence rate is

where $\mathcal{H}$ is the hypothesis distribution class and $\beta$ is any estimator of $\beta^{*}$. The minimax lower bound for sparse linear regression under LDP is well explored in Wang & Xu (2019) and Zhu et al. (2023).

The above result yields that for $d\gtrsim nm$, any attempt to LDP sparse linear regression is effortless, as the estimation error does not even converge. In fact, similar negative result also holds when $m$ is small yet larger than 1.

Proposition 2.3 shows that for $m\leq s^{*}\log d$, the error does dot converge to zero as $n$ grows. However, this is not the case for user-level LDP if $m$ is sufficiently large. The rigorous counterargument is by establishing upper bound in Theorem 3.6, which is $\mathcal{O}(s^{*2}/nm\varepsilon^{2})$. We explain why the bound fails to generalize. Its proof involves construction of a function class $\mathrm{P}_{Z}$ and a distribution of $Z$, such that the mutual information between $Z$ and private views $U_{1},\cdots,U_{n}$ is bounded from above and below. The former does not hold any more given $m$ samples, since the mutual information becomes larger exponentially in $m$. By carefully bounding the quantity, we establish the following lower bound for ULDP.

The result shows that any ULDP estimator admits an error scaling at least with $1/nm\varepsilon^{2}$. Thus, a possible improvement for ULDP over LDP lies in replacing $d$ with $s^{*}$.

In this section, we elucidate the steps for candidate variable selection. First, each user prepares a piece of information $v_{i}\in[d]$, indicating the variable is selected by user $i$ and probably belong to the true variable set. Then, a curator privately aggregates the information ${v}_{i}$s and outputs the candidate variables.

To formalize $v_{i}$, each user $i$ adopt a local selector $\mathcal{S}_{i}:(\mathcal{X}\times\mathcal{Y})^{m}\to[d]$. For each $i$, $\mathcal{S}_{i}$ can be any plug-in method and is chosen differently based on the constraints of sample size, computational power, and prior information, as long as it produces a good selection results described as follows.

(2) requires a lower bound on probability for the true variables to be selected. To induce such selectors, we consider first conducting a local variable selection using $(X_{i},y_{i})$ and uniformly sampling a $v_{i}$. The following proposition demonstrates that obtaining such a selector is feasible given mild assumptions on distribution $\mathrm{P}$, leveraging well-developed variable selection methods.

See Appendix B.1 for examples of precise algorithms and detailed proofs. (i) and (ii) are examples of sufficient conditions that are relatively easy to satisfy. They require mild correlations among covariates, a strong signal (minimum absolute value of $\beta^{*}$), and an adequate number of local samples. Similar conditions are standard in high-dimensional statistics (Fan & Li, 2001; Zhao & Yu, 2006). Though the lower bound of $a$ is considerable and will leads to a improved minimax rate in the non-private case (Ndaoud, 2019), it is not the key for ULDP sparse linear regression to be advantages over its LDP counterpart. This is because the function classes constructed in Wang & Xu (2019); Zhu et al. (2023) for lower bound proof are all covered by the assumptions. Additionally, the sample size requirement remains polynomial in $s^{*}$ and $\log d$, which is theoretically reasonable.

Given the local information $v_{i}$, we conduct a private voting to identify the frequently appeared variables $\{\widehat{v}^{1},\cdots,\widehat{v}^{s}\}$. Suppose we use the first $n/2$ users for identification, although the proportion is arbitrary and can be any constant. Considering the large size $d$ of the variable universe compared to number of available users, this task is closely related to the problem of heavy hitter detection (Bassily et al., 2020; Acharya et al., 2021). We solve the identification problem in standard manner (Bassily et al., 2020), while any tailored approach is adoptable. Specifically, we encode the $d$ variables into a binary string using $\lceil\log d\rceil$ bits. Next, we traverse a binary prefix tree from level $1$ to $\lceil\log d\rceil$ and eliminate nodes that cannot serve as prefixes of heavy hitters, namely those with frequencies lower than a certain threshold $\rho$. The key advantage of this method is its ability to identify frequent elements with frequencies above $\sqrt{n\log d\log n/\varepsilon^{2}}$, which overcomes the polynomial dependency on $d$ in LDP discrete density estimation (Kairouz et al., 2016; Duchi et al., 2018). The detailed procedure (HeavyHitter) is provided in Appendix B.2.

In the first part of Algorithm 1, we summarize the pipline for candidate variable selection. The following proposition demonstrates its effectiveness by establishing that, provided the existence of local good selectors, the curator can select a set of variables of size $s\asymp s^{*}$ containing the true variables. This property, known as perfect selection or consistent selection (Zhao & Yu, 2006; Belloni & Chernozhukov, 2013), plays a crucial role in the theoretical properties of subsequent operations.

Note that our scheme samples only one locally selected variable and disregards the others. This select-one-and-aggregate approach has been demonstrated to be as effective as if each user had only one variable (Zhu et al., 2020; Cohen et al., 2023). To fully utilize the information of the selected variables, we can leverage the set-value heavy hitters (Qin et al., 2016; Zhu et al., 2020; Wang et al., 2023). However, this only results in an improvement of $O(\sqrt{s^{*}})$ in the threshold, which is not our primary focus.

Given the selected variables $\{\widehat{v}_{1},\cdots,\widehat{v}_{s}\}$, the problem is reduced to low dimensional linear regression. Efficient algorithms and fundamental limits have been established (Duchi et al., 2018; Wang & Xu, 2019) for item-level LDP. Leveraging these algorithms, one can ignore all but one sample from each user and obtain an error bound depending polynomially on $s^{*}$ instead of $d$. However, we would like to explore the benefits brought by having multiple samples per user, as is addressed in the advanced research of ULDP.

We introduce necessary notations to define the learning problem on the selected subspace. Given Proposition 3.3, we assume the selected variables contain the true ones in the following analysis. Without loss of generality, let $(\widehat{v}_{1},\cdots,\widehat{v}_{s})=(1,\cdots,s)$. We put a hat over the quantities on the selected space. Let $\widehat{\mathrm{P}}$ be the marginal distribution on the selected space $\widehat{\mathcal{X}}\times\mathcal{Y}=\mathbb{R}^{s+1}$, where $\widehat{\mathrm{P}}_{\widehat{X}}=\mathrm{P}_{X^{1:s}}$. Define the data on selected space as $\widehat{X}_{i,j}={X}_{i,j}^{1:s}$ and $\widehat{X}_{i}=\{{X}_{i,j}^{1:s}\}_{j=1}^{m}$. The underlying coefficients becomes $\widehat{\beta}^{*}=\beta^{*1:s}$.