firrm : Statistical of Quantization
firrm : Statistical of Quantization
firrm : Statistical of Quantization
I. INTRODUCTION
-
certain impulse carrier [Fig. 2(a)l 141. -3T -2T -T 0 T 2T 3T
t
The spectrum of the original signal is repeated along the
frequency axis [Fig. 2(c)]. When the repeated spectra do not
overlap, the original spectrum [Fig. 2(b)] cart be restored, and
its inverse Fourier transform yields the origin(a1input signal. In Fkequency Domain
I X(jW)
other words, the sampled signal contains the iiame information
as the continuous-time signal, and it can be used for the same
purposes as the original signal. This statement in a precise
mathematical form is the sampling theorem.
11. QUANTIZATION
AS SAMPLING OF THE
PROBABILITY
DENSITYFUNCTION (PDF)
Quantization is generally less well understood than sam-
pling. The reason is that it is a nonlinear operation; therefore
most people believe that standard tools of linear system theory
cannot be applied to it. In fact, we will show how linear
system theory can be precisely used to analyze the effect of (C)
quantization on moments and other statistical properties of the Fig. 2. The Fourier transform of a time function, and the Fourier transform
signals. of its samples: (a) a time function being sampled; (b) symbolic representation
of Fourier transform of time function; and (c) symbolic representation of
Sampling discretizes time, and quantization discretizes am- Fourier transform of samples of time function.
plitude. One would expect that quantizaticn has a similar
effect on functions of the amplitude as sampling has on
functions of time. This recognition led Widrow to the study of probability density functions (PDF’s) and to the dlevelopment
Manuscript received April 24, 1995; revised October 30, 1995. This work of a statistical theory of quantization in the late 1950’s [5]-[7].
was sponsored by the National Science Foundation under Grant NSF IRI- The characteristics of a uniform quantizer are pictured in
9113491-A1, the Electric Power Research Institute under Grant 2DPM901,
the Fulbrighl Program, and the US-Hungarian Science and Technology Joint
Fig. 3(a), and a symbolic representation of quantization as an
Fund in cooperation with the Hungarian Academy of Sciences and the National operator is shown in Fig. 3(b). The quantizer input is x, and
Institute of Standards and Technology under Project 290. the quantizer output is XI. Quantization is an operation on
The authors are with the Department of Electrical Engineering, Information
Systems Laboratory, Stanford University, Stanford, CA 94305-4055 USA. signals that is represented as a “staircase” function. a nonlinear
Publisher Item Identifier S 0018-9456(96)02495-3. relation between x’ and x.
0018-9456/96$05.00 0 1996 IEEE
354 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 45, NO. 2, APRIL 1996
?-
39
2Q
,Ave;age slope = 1
Input output
; % % 2 2
2
-9
-2q
-39
Fig. 4. Formation of the PDF of the quantizer output x': area sampling.
input PDF within the bounds of each quantum box. Cutting Every step of the way in going from f z ( x ) to f z ' (x)involves
up the input PDF into strips as in Fig. 4, the area of each strip linear operations.
is compressed into an impulse in the center of the strip when The Fourier transform of the PDF is known in statistics as
forming the output PDF. This is like a sampling process, and the characteristic function, the CF. The input CF is
we call it area sampling.
Area sampling can be accomplished by first convolving the
input PDF f z ( z )with a uniform pulse
7 r Q
@,(U) = 0 for IuI > - = - (4)
4 2
then
I the CF of z can be derived from the CF of x’,and
the PDF of z can be derived from the PDF of z’.
The proof is straightforward from (3). The quantizing theo-
rem provides the condition for the output PDF of thie quantized
signal to contain all the information about the input PDF. In
other words, we established a one-to-one connection between
the statistiical descriptions of the input and output signals of
the quantizer.
The above considerations lead to another very important
consequence. By taking the central replica of the CF, or
equivalently, by interpolating the output PDF, we obtain not
the input PDF, but its convolution with a uniform PDF.
Therefore. the central replica of the CF is the product of the
CF of z and the CF of a uniform distribution.
The analogy between sampling - - and quantization is even
*=(U) sinc f ~ . , ( U- Q~ sine p(pr a ~ ) more profiound. When a signal is not band-limited, we usually
apply an anti-aliasing filter to it before sampling. The anti-
aliasing filter multiplies the spectrum by the trans Fer function
of the filter, which is zero outside the desired passband.
Similarly, in quantization we can find a way to rnultiply the
-2Q -aI 0 aI 24
CF by a desired function. A product of characteristic functions
corresponds to convolution-in the PDF domain, since CF’s
and PDF’:; are Fourier transform pairs. Convolution of PDF’s
I corresponds to addition of independent random variables.
aI 2Q
Therefore. we can limit the band of the CF by adding an
independent random variable with limited CF bandwidth to
the input signal. This auxiliary signal is called dither, well-
known in the practice of A/Dconversion and digital signal
(e) processing [8]-[ 111. This is a very important topic, but because
Fig. 6. Formulation of area sampling in the CF domain: (a) CF of n:; (b)
of the limiited space, we have to refer here to the literature for
+
CF of n, the sinc function; (c) CF of n: n; (d) the repetition of (c); and more detail.
(e) CF of d .
111. RECONSTRUCTION
OF THE INPUT PDF
is shown in Fig. 6(d), and the sum of the repetitions is shown It follows from the model described above that as long as
in Fig. 6(e). This is a sketch of the Fourier transform of the QT I is satisfied, the output and input PDF’s are uniquely
output PDF of Fig. 5(e). A general expression for the CF of related to leach other. Therefore, a crude histogram can be used
the quantizer output is for reconstruction of the input PDF. This is illustrated on age
distribution of the 1992 US census data [12], see Fig. 7. The
left-hand plots show histograms artificially made coarser than
the usual one-year resolution; the right-hand side plots show
the interpolated results superimposed on the bar g a p h of the
original census data. It is striking how good the interpolation
where sinc ( U ) = sin (U)/.. results are even with ten-year input resolution.’
Equation (3) clearly shows the repetition at integer multiples
of 9,the quantization frequency. This is analogous to the ‘The distribution has a significant jump at zero, and this makes the
sampling radian frequency, R = 27r/T, where T is the CF wide, violating QT I. For the calculation of the reconstruction, we
continued the histogram and the PDF by their mirror images in order to
sampling period. The sampling period is analogous to the avoid these problems. Sinc function interpolation was performed, followed by
quantization box size q. deconvolution of the rectangular pulse.
356 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 45, NO. 2, APRIL 1996
-enL
xqz-J X1
f” ( n ) from the moments of d.
Equation (7) will apply even when the replicas of Fig. 6(d)
overlap, as long as the overlap does not impact on the
derivatives at the origin. This leads to a second quantizing
++ theorem that applies to moments.
n
n -qP qI2 Quantizing Theorem II (QT II): If the CF of x is band-
(a) (b) limited so that
Fig. 8. Comparison of quantization with addition of independent noise: (a) 2T
quantization and noise addition, and (b) PDF of the noise. @,(U) =0 for Iul > -- E =Q - E, (8)
4
IV. MOMENTS with E positive and arbitrarily small, then the moments of IC
The moments of a random variable 5 , such as the mean, can be calculated from the moments of d.
mean square, mean cube, etc., can be determined by taking QT I and QT I1 were first proved by Widrow [5]. He has
derivatives of the CF at the origin. The kth moment is also shown that if QT I or QT I1 holds, the moments of the
quantized variable are equal to the moments of the sum of the
input variable and a uniformly distributed noise. This noise has
a mean of zero, a mean square of q2/12, a mean cube of zero,
a mean fourth of q4/80, etc. A rearrangement of these relations
One can verify this by differentiating ( 2 ) , making use of the
yields Sheppard’s famous corrections [ 131, [ 141, originally
definition
developed for grouped data under some smoothness conditions
E { x k }2 lm
00
x k f z ( x )dx. (6) on the PDF. The most right-hand terms (in parentheses) are
the Sheppard corrections
n
Sheppard’s corrections, can be approximated by a similar
negative power function of the amplitude A. and this yields a
simple expression of a minimum A (or a maximum q when
A is fixed) [15].
w 2 . X -q/2 q p-
The PDF of the quantization noise fv(x)can be computed in
the manner illustrated in Fig. 9. A given value of U results from Fig. 9. Construction of the PDF of quantization noise.
quantization of x falling at just the right places within all of the
quantization boxes. The probability of getting a given value
of U is the sum of probabilities from all of the quantization
boxes. The PDF of U may therefore be constructed by cutting
the PDF of x into strips, and stacking and adding them. It has
been shown [6], [7] that the PDF of the quantization noise will
be exactly uniform if either QT I or QT I1 is !satisfied.As such, v Quantization noise
quantization noise has zero mean and a meart square of q2/12.
The necessary and sufficient condition for the quantization Fig. 10. Quantization noise, the difference between the quantizer output and
its input.
noise to be uniform was developed by Siipad and Snyder
[ 161. The condition is satisfied when the CF is equal to zero at as q = cr. Similar results are obtained with finer quantization
27rl/q, I = fl,f 2 , . . . . This is a condition milder than QT 11. for other input PDF’s.
NOISE:THEPQN MODEL
IX. PSEUDO QUANTIZATION
VIII. CROSS-CORRELATIONBETWEEN
Refer <onceagain to Fig. 8(a), where quantization is con-
QUANTIZATIONNOISEAND THE QUANTIZER INPUT
trasted with the addition of independent uniformly distributed
Fig. 10 shows how one could measure quantization noise noise. The addition of independent noise and quantization
I/, defined by (10). It is of great interest to’ know the cross- are not the same, since the quantizer adds noise that is
correlation between the quantization noise ,and the quantizer deterministically related to the signal being quantized. But
input, to learn something about their relationship. It is clear, when the conditions for QT I or QT I1 are met, all moments
first of all, that the noise and the input are deterministically and joint moments correspond exactly for quantization and
related. For a given input, there is a definite output and a the addition of independent uniformly distributed noise. For
definite difference between output and input. Although the example, when QT I or QT I1 is satisfied,
quantization noise and the quantizer input are deterministically
related, it is a curious fact that under certain circumstances, the E{(z’)’”} +
= E { ( x @} (11)
input and noise are uncorrelated. It had been shown by Widrow E { V k } =E { d } (12)
[6], [7] that when either QT I or QT I1 is satisfied, quantization E { z V } = E{z”i}
noise is uncorrelated with the signal being quantized. These
= E{z”E{nl} (13)
conditions are met with Gaussian inputs to a very close
approximation even when the quantization step size is as large for all positive integers k and 1.
358 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 45,NO. 2, APRIL 1996
X. HIGH-ORDERSTATISTICAL
OF QUANTIZATION
DESCRIPTION
Refer now to Fig. 1. To describe the statistics of the P.11.2
multiple-sample quantizer input, a multidimensional joint PDF
would be required. These are high-order statistical descrip- Fig. 11. Correlation coefficient of the quantization noise as a function of the
correlation coefficient of the quantizer input. The input is Gaussian.
tions.
High-order forms of QT I and QT I1 exist. They can be
stated as follows.
Multidimensional Quantizing Theorem I (QT I): If the CF for every integer value of 11 and 12, except (11, 1 2 ) = (0, 0).
of a sequence of quantizer input samples is “band-limited” This condition is quite difficult to apply in practice; therefore
in N-dimensions, so that alternative formulations are of great importance (see below).
In practice, input CF’s are not exactly band-limited, and
the quantizing theorems apply only approximately. High-order
CF’s have some overlap with their repetitive parts, and this
impacts their moments. If the quantizer input is Gaussian
with q as big as 0, correlations among input samples as
for any IC E [I,NI (14)
high as 99% will cause correlations among corresponding
then quantization noise samples of only 1%. The quantization
the CF of xl, ... , X N can be derived from the CF of noise will be essentially white, having a flat spectrum and an
xi, . . . , xL, and impulsive autocorrelation function almost without regard to the
the PDF of 2 1 , . . . , X N can be derived from the PDF of autocorrelation function of the quantizer input. Fig. 11 shows
theoretical plots of correlation coefficients of quantization
Xi, . . . , xh.
noise samples versus correlation coefficients of corresponding
Multidimensional Quantizing Theorem II (QT II): If the CF of
quantizer input samples. Similar curves were derived by
xl,. . . , X N is band-limited in N-dimensions, so that
Widrow in 1956 [6].
@ z l ,... , Z N (Ul, ... , U N ) = 0 An approximate condition of whiteness was developed in
271.
[17] as
when /ukl> -- E =Q - E
4
for any k E [l,NI (15)
t
4’
2 Y’
Compressor
4- L Q Expande:r
-
8A
Nonlinear Uniform quantizer Inverse
function (“Hidden quantizer”) nonlinear
function
4A - gain = 1
Fig. 14. A model of a floating-point quantizer.
2A -
Fig. 12. Input-output staircase function for a floating-point quantizer with a Fig. 15, The input-output characteristic of the compressor,
3-bit mantissa, i.e., p = 3.
o.20i In
0.15
Jan. 1961 Section.
D. T. Sherwood, “Some theorems on quantization and an example
using dither,” in Con5 Rec. 19th Asilomar Con5 Circuits, Systems and
Computers, Pacific Grove, CA, Nov. 6-8, 1986, 86CH2331-7, pp.
207-212.
S. P. Lipshitz and R. A. Wannamaker, “Quantization and dither: A
theoretical survey,” J. Audio Eng. Soc., vol. 40, no. 5 , pp. 355-375,
0.05 t J -L May 1992.
P. Carbone et al., “Effect of additive dither on the resolution of ideal
quantizers,” IEEE Trans. Instrum. Meas., vol. 43, no. 3, pp. 389-396,
June 1994.
R. M. Gray and T. G. Stockham, Jr., “Dithered quantizers,” IEEE Trans.
hform. Theory, vol. 39, no. 3, pp. 805-812, May 1993.
Statistical Abstract of the United States. The National Data Book, 114th
-d. U.S. Dept. of Commerce, Economics and Statistics Administration,
Bureau of the Census, 1994.
W. F. Sheppard, “On the calculation of the most probable values of
frequency-constants, for data arranged according to equidistant divisions
of scale,” Proc. London Math. Soc., vol. 29, pp. 353-380, 1898.
A. Stuart and J. K. Ord, Kendall’s Advanced Theory of Statistics, Vol.
1, Distribution Theory, 6th ed. London: Edward Arnold-New York:
Wiley, 1994.
I. KollAr, “Bias of mean value and mean square value measurements
based on quantized data,” IEEE Trans. Instrum. Meas., vol. 43, no. 5 ,
pp. 733-739, Oct. 1994.
A. B. Sripad and D. L. Snyder, “A necessary and sufficient condition
for quantization errors to be uniform and white,” IEEE Trans. Acoust,
Speech, Signal Processing, vol. ASSP-25, no. 5 , pp. 442-448, Oct. 1917.
(b) (c) I. Kollh, “The noise model of quantization,” in Proc. 1st ZMEKO TC4
Symp. Noise in Electrical Measurements, Como, Italy, June 19-21, 1986;
Fig. 16. PDF of compressor output and of hidden quantization noise when
Budapest, OMIKK-Technoinform, 1987, pp. 125-129.
z is zero-mean Gaussian with 0 = lOOA: (a) f g ( y ) ; (b) fv(v) for p = 4
R. W. Hamming, “On the distribution of numbers,” Bell System Tech.
( q = A/8); and (c) f v ( v )for p = 8 ( q = A/128).
J., vol. 49, no. 8, pp. 1609-1625, Oct. 1970.
T. Kaneko and B. Liu, “On local roundoff errors in floating-point
arithmetic,” J. Assoc. Comp. Mach., vol. 20, no. 3, pp. 391-398, July
These and further results can be applied to scientific com- 1973.
plltations and floating-point signal processing, as floating- J. Kontro, K. Kalliojiirvi, and Y. Neuvo, “Floating-point arithmetic in
signal processing,” in Proc. IEEE Int. Symp. Circuits and Systems, San
point digital filters, floating-point FFT, and so on. Deriva- Diego, CA, May 10-13, 1992, 92CH3139-3, vol. 4, pp. 1784-1791.
tions, proofs, and applications will be given in a forthcoming A. Lacroix and F. Hartwig, “Distribution densities of the mantissa and
Prentice-Hall book entitled “Quantization Noise,” by Widrow exponent of floating-point numbers,” in IEEE Int. Symp. Circuits and
Systems, San Diego, CA, May 1992, pp. 1792-1795.
and KollBr [26]. Other useful and related references are B. Liu and T. Kaneko, “Error analysis of digital filters realized with
[27]-[35]. floating-point arithmetic,” Proc. IEEE, vol. 57, no. 10, pp. 1735-1747,
Oct. 1969.
A. V. Oppenheim and C. J. Weinstein, “Effects of finite register length
in digital filtering and the fast fourier transform,” Proc. IEEE, vol. 60,
XII. CONCLUSIONS no. 8, Aug. 1972, pp. 957-976.
A. B. Sripad and D. L. Snyder, “Quantization errors in floating-point
A brief survey of the statistical theory of quantization was arithmetic,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-
presented. The most important results were summarized, and 6, no. 5, pp. 456-463, Oct. 1978:
application of the theory to floating-point quantization was The Institute of Electrical and Electronics Engineers, “IEEE Standard for
binary floating-point nithmetic,” ANSVIEEE Standard 754-1985, New
presented. This theory is a very powerful tool to analyze York, Aug. 1985; “IEEE Standard for radix-independent floating-point
statistical properties of quantized variables and of estimators arithmetic,” ANSUIEEE Standard 854-1987, New York, Oct. 1987.
calculated from them. B. Widrow and I. Kollir, Quantization Noise. Englewood Cliffs, NJ:
Prentice-Hall, in preparation.
D. Bellan, A. Brandolini, and A. Gandelli, “Quantization theory in
REFERENCES electrical and electronic measurements,” in Instrumentation and Mea-
surement Tech. Con$, IMTC’95, Waltham, MA, Apr. 24-26, 1995.
C. E. Shannon, “Communication in the presence of noise,” Proc. IRE, W. R. Bennett, “Spectra of quantized signals,” Bell System Tech. J., vol.
vol. 47, pp. 10-21, 1949. 27, no. 3, pp. 446-472, July 1948.
A. J. Jerri, “The Shannon sampling theorem-Its various extensions T. A. C. M. Claasen and A. Jongepier, “Model for the power spectral
and applications: A tutorial review,” Proc. IEEE, vol. 65, no. 11, pp. density of quantization noise,” IEEE Trans. Acoustics, Speech, and
1565-1596, Nov. 1977. Signal Processing, vol. ASSP-29, no. 4, pp. 914-917, 1981.
R. J. Marks, 11, Introduction to Shannon Sampling and Interpolation T. Dobrowiecki, “Quantized error spectra at high frequencies for a
Theory. New York: Springer-Verlag, 1991 certain class of signals,” in Proc. 2nd IMEKO TC7 Symp. Application of
W. K. Linvill, “Sampled-data control systems studied through compari- Statistical Methods in Measurement, Leningrad, May 16-19, 1978, pp.
son of sampling with amplitude modulation,” AIEE Trans., vol. 70, pp. Dobrowiecki/l-8.
1779-1788, 1951. R. M. Gray, “Quantization noise spectra,” IEEE Trans. Information
B. Widrow, “ A study of rough amplitude quantization by means Theory, vol. 36, no. 6, pp, 1220-1244, Nov. 1990.
of Nyquist sampling theory,” Sc.D. thesis, Department of Electrical J. Katzenelson, “On errors introduced by combined sampling and
Engineering, MIT, June 1956. quantization,” IRE Trans. Automat. Contr, vol. AC-7, pp. 58-68, 1962.
~, “A study of rough amplitude quantization by means of Nyquist I. Kollh, “Statistical theory of quantization: Results and limits,” Pe-
sampling theory,” IRE Trans. Circuit Theory, vol. 3, no. 4, pp. 266-276, riodica Polytechnica Ser. Elect. Eng., vol. 28, no. 213, pp. 173-190,
Dec. 1956. 1984.
WIDROW et al.: STATISTICAL THEORY OF QUANTIZATION 361
[34] G. H. Robertson, “Computer study of quantizer output spectra,” Bell I s t v b Kollhr (M’87-SM’93) was born in Bu-
System Tech. J., vol. 48, no. 5 , pp. 2391-2403, 1969. dapest, Hungary in 1954. He graduated in electrical
[35] A. I. Velichkin, “Correlation function and spectral density of a quantized engineering from the Technical University of Bu-
process,” Telecommunications and Radio Engineering, Part 11: Radio dapest in 1977, and in 1985 received the degree
Engineering, pp. IQ-77, July 1962. “Candidate of Sciences” (the equivalent of Ph.D.)
from the Hungarian Academy of Sciences, and the
degree dr tech. from the Technical University of
Budapest.
From September 1993 to June 11995, he was a
Bernard Widrow (M’58-SM’75-F’76-LF’95) re- Fulbright scholar and visiting associate professor in
ceived the S B , S.M , and Sc D degrees from the the Department of Electrical Engineering, Stanford
Massachusetts Institute of Technology, Cambridge, University. He is associate professor of electrical engineering, Department of
in 1951, 1953, and 1956, reslpectively. Measurement and Instrument Engineering, Technical University of Budapest.
He was with MIT until he joined the faculty, His research interests span the areas of digital and analog signal processing,
Stanford University, Stanford, CA, in 1959, where measurement theory, and system identification. He has publlished about 50
he is now professor of electrical engineering He IS scientific papers, and is coauthor of the book Technology of Electrical
presently engaged in research and teaching in neu- Measurements, (L. Schnell, Ed., Wiley, 1993). He authored the Frequency
ral networks, pattern recognition, adaptive filtering, Domain Sy ytem Ident&ation Toolbox for Matlab.
adaptive control systems, and quantization theory.
He is associate editor of the journals Adaptive
Control and Signal Processing, Neural Networks, Information Sciences, and
Pattern Recognition, is coauthor with S. D. Stearns of Adaptive Signal
Processing (Prentice-Hall), and coauthor with E. Walach of Adaptive Inverse Ming-Chang Liu was born in Taipei, Taiwan in
Control (Prentice-HaIl). 1962. He received the B.S. degree from the National
Dr. Widrow is a member of the National Academy of Engineering, Taiwan University in 1984, and 1W.S. degree of
the American Association of University Professors, the Pattem Recognition mechanical engineering from Stanford University in
Society, Sigma Xi, and Tau Beta Pi. He is a Fellow of the American 1989. He is currently a Ph.D. candidate in electrical
Association for the Advancement of Science, and is past president of the engineering at Stanford University. His research
International Neural Network Society. Professor Widrow received the IEEE interests are in the areas of statistical signal pro-
Centennial Medal in 1984, and the IEEE Neural Networks Pioneer Medal cessing, quantization noise, and adaptive control.
in 1991. In 1986, he received the IEEE Alexander Graham Bell Medal for
exceptional contributions to the advancement of telecommunications.