-
Do Neural Networks Compress Manifolds Optimally?
Authors:
Sourbh Bhadane,
Aaron B. Wagner,
Johannes Ballé
Abstract:
Artificial Neural-Network-based (ANN-based) lossy compressors have recently obtained striking results on several sources. Their success may be ascribed to an ability to identify the structure of low-dimensional manifolds in high-dimensional ambient spaces. Indeed, prior work has shown that ANN-based compressors can achieve the optimal entropy-distortion curve for some such sources. In contrast, we…
▽ More
Artificial Neural-Network-based (ANN-based) lossy compressors have recently obtained striking results on several sources. Their success may be ascribed to an ability to identify the structure of low-dimensional manifolds in high-dimensional ambient spaces. Indeed, prior work has shown that ANN-based compressors can achieve the optimal entropy-distortion curve for some such sources. In contrast, we determine the optimal entropy-distortion tradeoffs for two low-dimensional manifolds with circular structure and show that state-of-the-art ANN-based compressors fail to optimally compress them.
△ Less
Submitted 9 September, 2022; v1 submitted 17 May, 2022;
originally announced May 2022.
-
On One-Bit Quantization
Authors:
Sourbh Bhadane,
Aaron B. Wagner
Abstract:
We consider the one-bit quantizer that minimizes the mean squared error for a source living in a real Hilbert space. The optimal quantizer is a projection followed by a thresholding operation, and we provide methods for identifying the optimal direction along which to project. As an application of our methods, we characterize the optimal one-bit quantizer for a continuous-time random process that…
▽ More
We consider the one-bit quantizer that minimizes the mean squared error for a source living in a real Hilbert space. The optimal quantizer is a projection followed by a thresholding operation, and we provide methods for identifying the optimal direction along which to project. As an application of our methods, we characterize the optimal one-bit quantizer for a continuous-time random process that exhibits low-dimensional structure. We numerically show that this optimal quantizer is found by a neural-network-based compressor trained via stochastic gradient descent.
△ Less
Submitted 10 February, 2022;
originally announced February 2022.
-
Principal Bit Analysis: Autoencoding with Schur-Concave Loss
Authors:
Sourbh Bhadane,
Aaron B. Wagner,
Jayadev Acharya
Abstract:
We consider a linear autoencoder in which the latent variables are quantized, or corrupted by noise, and the constraint is Schur-concave in the set of latent variances. Although finding the optimal encoder/decoder pair for this setup is a nonconvex optimization problem, we show that decomposing the source into its principal components is optimal. If the constraint is strictly Schur-concave and the…
▽ More
We consider a linear autoencoder in which the latent variables are quantized, or corrupted by noise, and the constraint is Schur-concave in the set of latent variances. Although finding the optimal encoder/decoder pair for this setup is a nonconvex optimization problem, we show that decomposing the source into its principal components is optimal. If the constraint is strictly Schur-concave and the empirical covariance matrix has only simple eigenvalues, then any optimal encoder/decoder must decompose the source in this way. As one application, we consider a strictly Schur-concave constraint that estimates the number of bits needed to represent the latent variables under fixed-rate encoding, a setup that we call \emph{Principal Bit Analysis (PBA)}. This yields a practical, general-purpose, fixed-rate compressor that outperforms existing algorithms. As a second application, we show that a prototypical autoencoder-based variable-rate compressor is guaranteed to decompose the source into its principal components.
△ Less
Submitted 8 June, 2021; v1 submitted 5 June, 2021;
originally announced June 2021.
-
Estimating Entropy of Distributions in Constant Space
Authors:
Jayadev Acharya,
Sourbh Bhadane,
Piotr Indyk,
Ziteng Sun
Abstract:
We consider the task of estimating the entropy of $k$-ary distributions from samples in the streaming model, where space is limited. Our main contribution is an algorithm that requires $O\left(\frac{k \log (1/\varepsilon)^2}{\varepsilon^3}\right)$ samples and a constant $O(1)$ memory words of space and outputs a $\pm\varepsilon$ estimate of $H(p)$. Without space limitations, the sample complexity…
▽ More
We consider the task of estimating the entropy of $k$-ary distributions from samples in the streaming model, where space is limited. Our main contribution is an algorithm that requires $O\left(\frac{k \log (1/\varepsilon)^2}{\varepsilon^3}\right)$ samples and a constant $O(1)$ memory words of space and outputs a $\pm\varepsilon$ estimate of $H(p)$. Without space limitations, the sample complexity has been established as $S(k,\varepsilon)=Θ\left(\frac k{\varepsilon\log k}+\frac{\log^2 k}{\varepsilon^2}\right)$, which is sub-linear in the domain size $k$, and the current algorithms that achieve optimal sample complexity also require nearly-linear space in $k$.
Our algorithm partitions $[0,1]$ into intervals and estimates the entropy contribution of probability values in each interval. The intervals are designed to trade off the bias and variance of these estimates.
△ Less
Submitted 18 November, 2019;
originally announced November 2019.
-
Irregular Recovery and Unequal Locality for Locally Recoverable Codes with Availability
Authors:
Sourbh Bhadane,
Andrew Thangaraj
Abstract:
A code is said to be a Locally Recoverable Code (LRC) with availability if every coordinate can be recovered from multiple disjoint sets of other coordinates called recovering sets. The vector of sizes of recovering sets of a coordinate is called its recovery profile. In this work, we consider LRCs with availability under two different settings: (1) irregular recovery: non-constant recovery profil…
▽ More
A code is said to be a Locally Recoverable Code (LRC) with availability if every coordinate can be recovered from multiple disjoint sets of other coordinates called recovering sets. The vector of sizes of recovering sets of a coordinate is called its recovery profile. In this work, we consider LRCs with availability under two different settings: (1) irregular recovery: non-constant recovery profile that remains fixed for all coordinates, (2) unequal locality: regular recovery profile that can vary with coordinates. For each setting, we derive bounds for the minimum distance that generalize previously known bounds to the cases of irregular or varying recovery profiles. For the case of regular and fixed recovery profile, we show that a specific Tamo-Barg polynomial-evaluation construction is optimal for all-symbol locality, and we provide parity-check matrix constructions for information locality with availability.
△ Less
Submitted 14 May, 2017;
originally announced May 2017.
