Glen G. Langdon,Jr.

An Introduction to Arithmetic Coding

Arithmetic coding is a data compression technique that encodes data (the data string) by creating a code string which represents a
fractional value on the number line between 0 and 1. The coding algorithm is symbolwise recursive; i.e., it operates upon and
encodes (decodes) one data symbol per iteration or recursion. On each recursion, the algorithm successively partitions an interval
of the number line between 0 and I , and retains one of the partitions as the new interval. Thus, the algorithm successively deals
with smaller intervals, and the code string, viewed as a magnitude, lies in each of the nested intervals. The data string is recovered
by using magnitude comparisons on the code string to recreate how the encoder must have successively partitioned and retained
each nested subinterval. Arithmetic coding differs considerably from the more familiar compression coding techniques, such as
prefix (Huffman) codes. Also, it should not be confused with error control coding, whose object is to detect and correct errors in
computer operations. This paper presents the keynotions of arithmetic compression coding by meansof simple examples.

1. Introduction
Arithmetic coding maps a string of data (source) symbols to a frequency (probability) of the events. The encoder accepts the
code string in such a way that the original data can be event and some indication of its relative frequency and gen-
recovered from the code string. The encoding and decoding erates the code string.
algorithms perform arithmetic operations on the code string.
One recursion of the algorithm handles onedata symbol. A simple model is the memoryless model, where the data
Arithmetic coding is actually a family of codes which share symbols themselves are encoded according to a single code.
the property of treating the code string as a magnitude. For a Another model is the first-order Markov model, which uses
brief history ofthe development of arithmetic coding, refer to the previous symbol as the context for the current symbol.
Appendix 1. Consider, for example, compressing English sentences. If the
data symbol (in this case, a letter) “q” is the previous letter,
Compression systems we wouldexpect the next letter to be “u.” The first-order
The notion of compression systems captures the idea that data Markov modelis a dependent model; we have a different
may be transformed into something which is encoded, then expectation for each symbol (or in the example, each letter),
transmitted to a destination, then transformed back into the depending on the context. The context is, in a sense, a state
original data. Any data compression approach, whether em- governed by the past sequence of symbols. The purpose of a
ploying arithmetic coding, Huffman codes, or any other cod- context is to provide a probability distribution, or statistics,
ing technique, has a model which makes some assumptions for encoding (decoding) the next symbol.
about the data and the events encoded.
Corresponding to the symbols are statistics. To simplify the
The code itself can be independent of the model. Some discussion, consider a single-context model, i.e., the memory-
systemswhich compress waveforms ( e g , digitizedspeech) less model. Data compression results from encoding the more-
may predict the next valueand encode the error. In this model frequent symbols with short code-string length increases, and
the error and not the actual data is encoded. Typically, at the encoding the less-frequent events with long code length in-
encoder side of a compression system, the data to be com- creases. Let e, denote the occurrences of the ith symbol in a
pressedfeed a model unit. The model determines 1) the data string. For the memoryless model and a given code, let 4
event@) to be encoded, and 2) the estimate of the relative denote the length (in bits) of the code-string increase associated

0 Copyright 1984 byInternational Business Machines Corporation. Copying in printed form for private use is permitted without payment of
royalty provided that ( 1 ) each reproduction is done without alteration and (2) the Journal reference and IBM copyright notice are included on the
first page. The title and abstract, but no other portions, of this paper may be copied or distributed royalty free without further permission by
computer-based and other information-servicesystems. Permission to republish any other portion of this paper must be obtained from the Editor. 135

IBM J. RES. DEVELOP. VOL. 28 NO. 2 MARCH I 984 GLEN G . LANGDON. JR.
 Huffman
Table 1 Example code. Encoder
The encoder accepts the events to be encoded and generates
Symbol Codeword
Probability p Cumulative the code string.
(in
binary) probability P

a 0 .loo .Ooo The notionsof model structure andstatistics are important

b 10 ,010 .loo because they completelydetermine theavailable compression.
C 110 .oo1 . I 10
d 111 .oo 1 . I 11 Consider applications where the compression model is com-
plex, i.e., has several contexts anda need to adapt to the data
string statistics. Due to theflexibility of arithmetic coding, for
such applications the “compression problem” is equivalent to
the “modelingproblem.”
with symbol i. The code-string length corresponding to the
data string is obtained by replacing each data symbol with its Desirable properties of a coding method
associated length and summing thelengths: We now list some properties for which arithmetic coding is
amply suited.
c Cr4.
I

For most applications, we desire thejrst-infirst-out(FIFO)

If 4 is large for data symbols of high relative frequency (large
property: Events are decoded in the same order as they are
values of c,), the given code will almost surely fail to achieve
encoded. FIFO coding allows for adapting to the statistics of
compression. The wrong statistics (a popular symbol with a
the data string. With last-in jirst-out (LIFO) coding, the last
long lengthe )lead to a code stringwhich may have more bits
event encoded is the first event decoded, so adapting is diffi-
than the original data. For compression it is imperative to
cult.
closely approximate the relative frequency of the more-fre-
quent events. Denote the relative frequency of symbol i as p,
We desire nomorethan asmall storage buffer atthe
where p, = cJN, and N is the total number of symbols in the
encoder. Once events are encoded,we do notwant the encod-
data string. If we use a fixed frequency for each data symbol
ing of subsequent events to alter what has already been gen-
value, the best we can compress (accordingto ourgiven model)
erated.
is to assign length 4 as -log pi. Here, logarithms are taken to
the base 2 and the unitof length is the bit. Knowing the ideal
Theencoding algorithmshould be capableofaccepting
length for each symbol, we calculate the ideal code length for
successive events from different probabilitydistributions.
the given data string and memoryless model by replacing each
Arithmetic coding has this capability. Moreover,the code acts
instance of symboli in the datastring by length value -log pt,
directly on the probabilities, and can adapt “on the fly” to
and summing thelengths.
changing statistics. TraditionalHuffman codes require the
design of a different codeword set for different statistics.
Let us now review the componentsof a compressionsystem:
the model structure for contexts andevents, the statistics unit
for estimation of the event statistics, and theencoder.
An initial view of Huffman and arithmetic codes
We progress to a very simple arithmetic code by first using a
Model structure prefix (Huffman) code asan example. Our purpose is to
In practice, the model is a finite-state machine which operates introducethe basic notions of arithmetic codesina very
successively on each data symbol and determines the current simple setting.
event to be encoded and its context (i.e., which relative
frequency distribution applies to the current event). Often, Considerafour-symbolalphabet,for which the relative
eachevent is the data symbol itself, but the structure can frequencies 4, i, i, and Q call for respective codeword lengths
define other eventsfrom which thedata stringcould be of 1, 2, 3, and 3. Let us order thealphabet {a, b,c, d) according
reconstructed. For example, one could define an event such to relative frequency, and use the code of Table 1. The
as the runlength of a succession of repeated symbols, i.e., the probability column has the binary fraction associated with the
number of times the currentsymbol repeats itself. probability corresponding to the assigned length.
Statistics estimation
The estimation method computes the relative frequency dis- The encoding for the data string “a a b c” is 0.0. IO. 110,
tribution used for each context.Thecomputation may be where “ . ” is used as a delimiter to show the substitution of
performed beforehand, or may be performed during the en- the codeword for the symbol. The code also has the prefix
coding process, typically by a counting technique. For Huff- property (no codeword is the prefix of another). Decoding is
man codes, the event statistics are predetermined by the length performed by a matching or comparison process starting with
136 of the event’s codeword. the first bit of the code string. For decodingcodestring

GLEN G . LA N G W N , JR. IBM J. RES. DEVELOP, VOL. 28 - NO. 2 MARCH 1984

00 101 10, the first symbol is decoded
as “a” (the only codeword 0 ,100 ,110 Ill I
I I I
beginning with 0).We remove codeword 0, and the remaining c d
4
a b
code is 0101 10. The second symbol is similarly decoded as
“a,” leaving 101 IO. For string 101 10, the only codeword Figure 1 Codewords of Table 1 as points on unit interval.
beginning with 10 is “b,”so we are left with 1 I O for c.

We have described Table 1 in terms of Huffman coding. a b c d

We now present an arithmetic coding view, with the aid of 0 100 ,110 Ill I

Figure 1. We relate arithmetic coding to the process of sub- I a ,010 I h I C I I I I

dividing the unit interval, and we make twopoints:

Point I Each codeword (code point) is the sum of the proba-
bilities of the preceding symbols.
Point 2 The width or size of the subinterval to the right of Figure 2 Successive subdivision of unit interval for code of Table 1
each code pointcorresponds to theprobability of the and data string “a a b . . ..”
symbol.

We havepurposelyarranged Table 1 with the symbols

.lOOOOO.. .. We can decode by magnitude comparison;if the
ordered according to their probability p , and the codewords
code string is less than 0.1, the first data symbol must have
are assigned in numerical order. We now view the codewords
been “a”
as binary fractional values (.OW, .loo, .110 and .11 I). We
assume that the reader is familiar with binary fractions, i.e.,
Figure 2 shows how the encoding process continues. Once
that $,i, and are respectively represented as .I , .O I , and . 1 1 1
“a” has been encodedto [O,. l), we next subdivide this interval
in the binary number system. Notice from the construction
into the same proportions as theoriginal unit interval. Thus,
of Table 1, and refemng to thepreviously stated Point I , that
the subinterval assigned to the second “u” is [O,.Ol). For the
each codeword is the sumof the probabilities ofthe preceding
third symbol, we subdivide [0,.01), and the subinterval be-
symbols. Inother words, eachcodeword is a cumulative
longing to third symbol “6” is [.OO l ,.OO l l). Note that each of
probability P.
the two leading Os in the binary representation of this interval
comes from the codewords (Table 1) of the two symbols “a”
Now we view the codewords as points (or code points) on which precede the “b.” Forthefourth symbol ”c,” which
the number line from 0 to 1, or the unit interval, as shown in follows ‘‘a a b,” the corresponding subinterval is
Fig. 1. The four code points divide the unit interval into four [.0010110,.0010111).
subintervals. We identify each subinterval with the symbol
corresponding to its leftmost point. For example, the interval In arithmetic coding we treat thecode points, which delimit
for symbol “a” goes from 0 to . I , and for symbol “d” goes the interval partitions, as magnitudes. To define the interval,
from . I 1 1 to 1.O. Note also from the construction of Table 1, we specify 1) the leftmost point C, and 2) the interval width
and refemng to the previous Point 2, that the width or size of A . (Alternatively one can define the interval by the leftmost
the subinterval to the right of each code point corresponds to point and rightmost point, or by defining the rightmost point
the probability of the symbol. The codeword for symbol “a” and the available width.) Width A isavailable forfurther
has $ the interval, the codeword for“b”(. 100) has the interval, partitioning.
and “c” (. 1 IO) and “d”(. 1 11) each have Q of the interval.
We nowpresent somemathematicsto describe what is
happening pictorially in Fig. 2. From Fig. 2 and its description,
In the example data, the first symbol is “a,” and the corre-
we see that there aretwo recursive operations needed to define
sponding interval on the unit interval is [O,.l). The notation
the current interval. For encoding, the recursion begins with
“[0,.1)” means that 0 is includedin the interval, and that
the “current” values of code point C and available width A ,
fractions equal to orgreater than 0 but less than . 1 are in the
and uses the value of the symbol encoded to determine“new”
interval. The interval for symbol “b” is [.l,.l IO). Note that .1
values of code point C and width A . At the endof the current
belongs to the interval for “b” and not for “a” Thus, code
recursion, and before the next recursion, the “new” values of
strings generated by Table I which correspond to continua-
code point and width become the “current” values.
tions of data strings beginning with “a,” when viewed as a
fraction, never equalor exceed the value 0.1. Data string New code point
“ a d d d d d . . . ” is encodedas “0111111~..,”which when The new leftmost point of the new interval is the sum of the
viewed as afraction approaches butnever reaches value current code point C, and the product of the interval width 137

IBM J . RES. DEVELOP. VOL. 28 NO. 2 MARCH 1984 GLEN G. LANGDON, JR.
 0 ,001 1001 Ill I retain the important techniqueof the double recursion. Con-

-
I 4
I
, d I a I h I (
I
I sider thearrangement of Table 2. The “codeword”corre-
Id1 a I h I c I sponds to the cumulative probability P of the preceding sym-
1 - I
Id1 a I b ICI bols in the ordering.
I...l
The subdivision of the unit interval for Table2, and for the
Figure 3 Subdivision of unit interval for arithmetic code of Table 2 data string “ a a b,” is shown in Figure 3. In this example, we
and data string“a a b. . ..” retain Points 1 and 2 of the previous example, but no longer
have the prefix property of Huffman codes. Compare Figs. 2
Table 2 Arithmeticcodeexample. and 3 to see that the interval widths are the same but the
locations of the intervals have been changed in Fig. 3 to
Symbol Cumulative Symbol Length conform with the new ordering in Table 2.
probability P probability p
d .ooo .oo1 3 Let usagaincode the string “a a b c.” This example
b .oo1 .010 2 reinforces thedouble recursionoperations, where the new
a .o I 1 .IO0 1 values become the current values for the next recursion. It is
C .111 .oo1 3
helpful to understand the arithmeticprovided here, using the
“picture” of Fig. 3 for motivation.

W of the current interval and the cumulative probability P, The first “a” symbol yields the code point .O 1 1 and interval
for the symbol i being encoded: [.O 1 1,.1 1 l), as follows:
New C = Current C + ( A X Pi). First symbol ( a )
For example, after encoding “a a,” the current code point C C N e w c o d e p o i n t C = O + I X(.011)=.011.
is 0 and theinterval width A is .O 1. For “ a a b,” the new code (Current code point plus current width A times P.)
point is .OO I , determined as0 (current code point C), plus the A : New interval width A = 1 X (.1) = . l .
product (.Ol) X (.loo). The factor on theleft is the width A of (Current width A times probability p . )
the current interval, and the factor on the right is the cumu- In the arithmeticcoding literature, we have called the value A
lativeprobability P for symbol “b”; see the“Cumulative X P added to the oldcode point C, the augend.

probability” column of Table 1.

The second “a” identifies subinterval [.1001,.1101).
New interval width A
The width A of the current interval is the product of the Second symbol ( a )
probabilities of the data symbols encoded so far. Thus, the C: New code point = .011 + .1 X (.011) =
new interval width is .O 1 1 (current code point)
New A = Current A X Pi, .0011 (current width A times P,or augend)

where thecurrent symbol is i. For example,afterencod- .lo0 1. (new code point)

ing “a a b,” the interval width is ( . I ) X ( . I ) X (.Ol), which is A: New interval width A = . 1 X (.I) = .O 1.
.om I . (Current width A times probability p.)
Now the remaininginterval is one-fourththe width of the unit
In summary,we can systematically calculate the next inter- interval.
val from the leftmost point C and width A of the current
interval, given the probability p and cumulativeprobability P For the third symbol, “6,” we repeat the recursion.
values inTable I for the symbol to be encoded. The two
operations (new codepoint andnew width) thus forma double Third symbol ( 6 )
recursion. This double recursion is central to arithmetic cod- C: New code point = .lo01 + .01 X (.001) = .10011.
ing, and this particular version is characteristic of the class of .lo01 (currentcodepoint C )
FIFO arithmetic codes which use the symbolprobabilities .OOOO 1 (current width A times P, or augend)
directly. -
,1001I (new code point)
TheHuffman codeof Table I correspondsto a special A : New interval width A = .01 X (.01) = .0001.
integer-length arithmetic code. With arithmetic codes we can (Current width A times probability p.)
rearrange the symbols and forsake the notion of a k-bit code- Thus, following the codingof “ a a b,” the interval is
138 word for a symbol corresponding to a probability of 2-k. We [.loo1 1,.10101).

GLEN G. LANGDON, JR. IBM J. RES. DEVELOP. VOL. 28 NO. 2 MARCH 1984
 To handle the fourthletter, “c,” we continue as follows. . I O 1001 I lies in [.O 1 1,. I IO), which is a’s subinterval. We can
summarize this step asfollows.
Fourth symbol (c)
C: New code point = .IO011 + .0001 X (.Ill) Step I : Decoder C comparison Examine the code string and
= .1010011. determine the interval in which it lies. Decode the symbol
.IO01 1 (current code
point) corresponding to thatinterval.
,0000I 1 1 (current width A times P, or augend)
Since the second subinterval code pointwas obtained at the
. I O 1001 1 (new code point) encoder by addingsomethingto .011, we can prepare to
A : New interval width A = .0001 X (.001) = .0000001. decode the second symbol by subtracting .011 from the code
(Current width A times probability p.) string: .IO1001 1 - .011 = .0100011. We then have Step 2.

Carry-overproblem Step 2: Decoder C readjust Subtract from the codestring the

The encoding of the fourth symbol exemplifies a small prob- augend value of the code point for thedecoded symbol.
lem, called the carry-over problem. After encoding symbols
Also, since the values for the second subinterval were ad-
“a,” “a,”and “b,” each having codewords of lengths1, 1, and justed by multiplying by .I in the encoder A recursion, we
2, respectively, in Table I , the first four bits of an encoded can “undo” that multiplication by multiplying the remaining
string using Huffman coding would not change. However, in
value of the codestring by 2. Our code string isnow .IO000 1 I .
this arithmeticcode, the encoding of symbol “c” changed the
In summary, we have Step 3.
value of the third code-string bit. (The first three bits changed
from . I O 0 to .I01 .) The change was prompted by a carry-over, Step 3: DecoderC scaling Rescale thecode C for direct
because we are basically adding quantities to thecode string. comparison with P by undoingthe multiplication for the
We discuss carry-over control later on in thepaper. value A .

Code-string termination Now we can decode the second symbol from the adjusted
Following encoding of “a ab c,” thecurrent interval is codestring .IO001 1 by dealing directly with the values in
[.1010011,.1010100). Ifwe were to terminate the code string Table 2 and repeating Decoder Steps 1, 2, and 3.
at this point (no more data symbols to handle), any value
equal to or greater than ,101001 1, but less than .1010100, Decoder Step 1 Table 2 identifies “a” as the second data
would serve to identify the interval. symbol, because the adjusted code string is greater than .01 I
(codeword for “a”)but less than . I 1 I (codeword for “c”).
Let us overview the example. In our creation of code string
Decoder Step 2 Repeating the operation of subtracting .O I I ,
.1010011, we in effect added properly scaled cumulative prob-
we obtain
abilities P, called augends, to the code string. For the width
recursion on A , the interval widths are, fortuitously, negative .100011 - .01I = .001011.
integral powers of two, which can be represented as floating Decoder Step 3 Symbol “a” causesmultiplication by .I at
point numbers with one bit of precision. Multiplication by a the encoder, so the rescaled code string is obtained by doubling
negative integral power of two may be performed by a shift the result of Decoder Step 2:
right. The code string for“a a bc” is the result of the following
sum of augends, which displays the scaling by a right shift: .01011.

.o 1 I The third symbol is decoded as follows.

01 1 Decoder Step 1 Referring to Table 2 to decode the third
00 1 symbol, we see that .O I O 1 1 is equal to or greater than ,001
111 (codeword for “b”)but less than ,011 (codeword for “a”),and
.101001 I symbol “b” is decoded.
Decoding Decoder Step 2 Subtracting out .00 1 we have .OO 1 1 1:
Let us retain code string .lOlOOl 1 and decode it. Basically,
the code string tells the decoder what the encoder did. In a .01011 - ,001 = ,0011 I .
sense, the decoder recursively “undoes” the encoder’s recur- Decoder Step 3 Symbol “b”caused the encoder to multiply
sion. If, for the first data symbol, the encoder had encoded a by .O 1, which is undone by rescaling with a 2-bit shift:
“b,” then (referring to the cumulative probability P column
of Table 2), the code-string value would be at least ,001 but ,00111 becomes . I 1 I .
less than .011. For encoding an “a,” the code-string value To decode the fourth and last symbol, Decoder Step 1 is
would be at least .O 1 1 but less than . 1 1 1. Therefore, the first sufficient. The fourth symbol is decoded as “c,” whose code
symbol of the data string must be “a” because code-string point corresponds to the remainingcode string. 139

IBM J . RESDEVELOP. VOL. 28 - NO. 2 MARCH 1984 GLEN G . L ANGDON. JR.

 Symbol
- Codeword well as compression codes benefit from the code tree view.
U 0
b 10 Again, the code alphabet is the binary alphabet {O, 11. The
c II code-string tree represents the set of all finite binarycode
(a) strings, denoted {O,l)*.

We illustrate a conventional Huffman codein Figure 4,

which shows the code-stringtreefor mappingdata string
s = “b a a c” to a binary code string. Figure 4(a) shows a
codewordtable for data symbols “a,” “b,” and “c.” The
encoding proceeds recursively. The first symbol of string s,
“b,” is mapped to code string 10, “b a” is mapped to code
string 100, and so on, asindicated in Fig. 4(b). Thus the depth
of the code tree increases with each recursion. In Fig. 4(c) we
highlight the branches traveled to demonstrate that the coding
process successively identifies the underlinednodes inthe
code-string tree. The root of the codeword tree is attached to
the leaf at the current depth of the tree, as per the previous
data symbol.

At initialization, the available code space ( A ) is a set {O,l )*,

which corresponds to the unit interval. Following the encoding
of the first data symbol b, we identify node 10 by the path
from the root. The depth is 2. The current code space is now
all continuations of code string 10. We recursively subdivide,
or subset, the current code space. A property of prefix codes
is that a single node in the codespace is identified as theresult
of the subdivision operation. Intheunit intervalanalogy,
prefix codes identify single points on the interval. For arith-
metic codes, we can view the code as mapping a data string
IC)
to an interval of the unit interval, ,as shown in Fig. 3, or we
Figure 4 The code-string tree of a prefix code. Example code, image can view the result of the mapping asa set of finite strings, as
of data string is a single leaf: (a) code table, (b) data-string-to-code- shown in Fig. 4.
string transformation,(c) code-string tree.

2. A Binary Arithmetic Code (BAC)

We have presented a view of prefix codes as the successive
A general view of the coding process application of a subdivision operation on the code space in
We have related a special prefix code (the symbols were order to show that arithmetic coding successively subdivides
ordered with respect to their probability) and a special arith- the unit interval. We conceptually associate the unit interval
metic code (the probabilities were all negative integral powers with the code-string tree by a correspondence between the set
oftwo) by picturing theunit interval. The multiplication of leaves of the code-string tree at tree depth D on one hand,
inherent in the encoder width recursion for A , in the general and the rational fractions of denominator 2D on the other
case, yields a new A which has a greater number of significant hand.
digits than the factors. In our simple example, however, this
multiplication did notcause the required precision to increase We teach the binary arithmetic coding (BAC) algorithm by
with the length of the code string because the probabilities p means of an example. We have already laid the groundwork,
were integral negative powers of two. Arithmetic coding is since we follow theencoderand decoder operationsand
capable ofusing arbitrary probabilities by keeping the product general strategy of the previous example. See [ 2 ] for a more
to a fixed number of bits of precision. formal description.

A key advance of arithmetic coding was to contain the The BAC algorithm may be used for encoding any set of
required precision so that the significant digits of the multi- events,whatever the original form, by breaking the events
plications do not grow witb the code string. We can describe down for encoding into a succession of binary events. The
the use of a fixed number of significant bits in the setting of a BAC accepts this succession of events and delivers successive
140 code-string tree. Moreover, constrained channel codes [ 11 as bits of the code string.

GLEN G L ANGDON. JR IBM J. RES. DEVELOP. VOL. 28 NO. 2 MARCH 1984

 The encoder operates on variable MIN, whose values are T The BAC coder successively splits the width or size of the
(true) and F (false). The name MIN denotes “more probable available code space A , or current interval, into two subinter-
in.” If MIN is true (T), the event to be encoded is the more vals. The left subinterval is associated with F and the right
probable, and if MIN is false (F), the event to be encoded is subinterval with T. Variables C and A jointly describe the
the less probable. The decoder result is binary variable MOUT, current interval as, respectively, the leftmost point and the
of values T and F, where MOUT means “moreprobable out.” width. As with the initial code space, the current interval is
Similarly, at the decoder side, output value MOCJT is true (T) closed on the left and open on theright: [C,C+ A ) .
only when the decoded event is the more probable.
In the BAC, not all intervalwidths are integral negative
In practice, data to be encoded are not conveniently pro- powers of two. For example, where p of event F is 4, the other
vided to us as the “more” or “less” probable values. Binary probability for T must be i.For the width associated with T
data usually represent bits from the real world. Here, we leave of j , we have more than one bit of precision. The product of
to a statistics unit the determinationof event values T or F. probabilities greater than f can lead to a growing precision
problem. We solve the problem by representing space A with
Consider, for example, a black and white image of two- a floating point number to a fixed precision. We introduce
valued pels (pictureelements) which hasaprimary white variable E for the exponent, which controls the “data han-
background. For these data we associate the instances of a dling” and “shifting” aspect of the algorithm. We represent
white pel value to the “moreprobable” value (T) and a black variable A in floating point with the most significant bit of A
pel value into the“less probable” value (F). The statistics unit in position E from theleft. Thus theleading I-bit ofthe binary
would thus have an internal variable, MVAL, indicating that representation of A has value 2-”. For example, if A =
white maps toT. On the other hand,if we had an image with 0.00101 I , E has value 3, 2? is 0.001, and A is 1.01 1.
a black background, the mapping of values black and white
would be respectively to values T and F (MVAL is black). In In the encoder A recursion of the example of Table 2, the
a more complex model, if the same black and white image width is determined by a multiplication. In the simple BAC
had areas of white background interspersed with neighbor- algorithm, the smaller width is determined by the value SK,
hoods of black, the mappingof pel values black/white to event as in Eq. ( I ) , which follows. The other width is the difference
values F and T could change dynamically in accordance with between the current width and the smaller width, as in Eq.
the context (neighborhood) of the pel location. In a black (2), which follows. No multiplication is needed.
context, the black pel would be value T, whereas in the context
of a white neighborhood the black pel would be value F. The current interval is split according to the skew value SK
as follows. If SK is 1, the interval is split nearly in half, and if
The statistics unit must determine the additional informa- SK is 12, the interval is split with a very small subinterval
tion as to by how much value T is more popular than value assigned to F. Note that we roughly subdivide thecurrent
F. The BAC coder requires us to estimate therelative ratio of interval in a way which corresponds to therelative frequency
F to the nearest power of 2; does F appear 1 in 2, or I in 4, of each event. Let W(F) and W(T) be respectively the subin-
or 1 in 8, etc., or 1 in 4096? We therefore have 12 distinct terval widths assigned to F and to T. Specifically,
coding parameters SK, called skew, of respective index 1
through 12, to indicate the relative frequency of value F. In a W(F) = 2-(E+SK), (1)
crude sense, we select one of 12 “codes” for each event to be
encoded or decoded. By approximating to 12 skew values, with the remainder of the interval width A assigned to T:
instead of using a continuum of values, the maximum loss in
coding efficiency is less than 4 percent of the original file size W(T) = A - 2-(E+SK), (2)
at probabilities falling between skew numbers 1 and 2. The
loss at higher skew numbers is even less; see [2]. We can summarize thehandling of an event (value T or F)
in the BAC algorithm in three steps. The first and second steps
In what follows, our concern is how to code binary events correspond to the A and C recursions described earlier. The
after the relative frequencies have been estimated. third step is a “data handling” or scaling step which we have
ignored in the earlier examples. Let s denote thestring ofdata
symbols already encoded,and let notation C(s); A($),and E($)
The basic encoding process respectively denote the values of variables C, A , and E follow-
The double recursion introduced in conjunction with Table 2 ing the encoding of the data string. Now, after handling the
appears in the BAC algorithm as a recursion on variables C next event T, let the new values of variables C, A , and E be
(for code point) and A (for available space). The BAC algo- respectively denoted C(s,T), A(s,T), and E(s,T).
rithm is initialized with the code space as the unit interval
[O,l) from value 0.0 to value 1.0, with C = 0.0 and A = 1.0. Sfepsfor encoding of next event 141

IBM J. RES. DEVELOP. VOL. 28 NO. 2 MARCH 1984 GLEN G. LANGDON, JR.
 Table 3 Exampleencoding-refining the interval.

Event MIN SK E W(F) C A

(value) (skew) (A’s lead(interval
Os) pt) (least
(F width) A)

Initial - - 0 - 0.o000oo 1. m 0
1 T 3 0 .001 0.001o00 0.1 1 1000
2 T 1 1 .o1 0.01 lo00 0.101o00
3 F 1 1 .o 1 0.01 lo00 0.01oooo
4 T 1 2 .oo1 0 . 1 m 0.001o00

c-c
0 ,001 1 For Event I , SK is3 and E is 0. For Step 1, the width
r I 1

Y
1
1
/
associated with the value F, W(F), is 2” or 0.001. W(T) is
F T what is left over, or 1.000 - 0.001 = 0. I 1 1. See Figure 5.
S u b d ~ v i s ~ opoint
n Relative to Step2, Eq. (3), the subdivision point is C W(T) +
+
or 0 .OO 1 = .001. Since the binaryvalue is T and therelative
Figure 5 Interval splitting-subdivision for Event 1, Table 3.
frequency of the T event is equal to orgreater than 4,we keep
the larger (rightmost) subinterval. Refemng to Fig. 5 , we see
that the new values of C and A which describe the interval are
0 Oll .I
Width ,101
101
h
. I
now C(T) = 0.001 and A(T) = W(T) = 0.11 1. For Step 3, we
Y- note that A has developed a leading 0, so E = 1.
I
L
I I
1
(a)
t Subdlvlslun polnt For Event 2 of Table 3, the skew SK is 1 and E is now I ,
so W(F)i~2-(~+~)or0.01. W(T)isthus0.111 -0.010=0.101.

0
I
-
,011
L
Width 010
IO1
\
J
I
J
The subdivision point of the current interval is C + W(F), or
0.01 1. Again, the event value is T, so we keep the rightmost
part. The new value of C is the subdivision point 0.0 1 1, and
the new value ofA is W(T) or 0,101. Theleading 1-bit position
(b) of A has not changed, so E is still 1.
Figure 6 Intervalsplitting-subdivision for Event 3, Table 3: (a)
Current interval at endof Event 2 and subdivision point. (b) Current For Event 3 of Table 3, see Figure 6, which displays current
interval following encoding of Event3. interval [.011,1) ofwidth .101. The smaller width W(F) is
2-(’+’) or .01. We add this value to C to obtain
C + W(F), or subdivision point .101. See Fig. 6(a). Refemng
Step I Given skew SK and E (the leading Os of A ) , subdivide
to Event 3 of Table 3, the value to encode is F, so we must
the current width as in Eqs. ( I ) and (2).
now keep the left side of the subdivision. By keeping the F
Step 2 Given the event values (T or F), C, W(T), and W(F), subinterval, the value of C remains at .01I and A becomes
describe the new interval: W(F) or 0.01. Available width A has a new leading 0, so E
becomes 2. The resulting interval is shown in Fig. 6(b).
If T: C(s,T) = C(s) + W(F) and A(s,T) = W(T). (34
If F: C(s,F) = C(s) and A(s,F) = W(F). (3b) Separation of data handlingfrom the arithmetic
Arithmetic codes generate the code string by adding a sum-
Step 3 Given the new value of A , determine the new value
mand (called augend in the arithmetic coding literature) to
of E
the current code string and possibly shifting the result. The
If T: If A(s,T) < 2-€(”),then E(s,T) = E($) + 1; summation operation creates a problem called the carry-over
problem. We can, in thecourse of code generation, shift out a
otherwise E(s,T) = E($).
long string of 1s from the coding process. An addition could
If F: E(s,F) = E(s)+ SK. propagateacarry into the longstring of Is, changing the
values of higher-order bits until a 0 is converted to a 1 and
We continue the discussion by an example, where we en- stopping thecurry chain.
code the four-event string T, T, F, T under respective skews
3, I , 1, 1. Theencoding is described by Table 3, andthe In this section we show how the arithmetic using A and C
142 following description accompanies thistable. can be separated fromthe carry-overhandling anddata-

GLEN G. LANGDON. JR IBM J. RES. DEVELOP. VOL. 28 NO. 2 MARCH 1984

buffering functions. The scheme of Table 3 assumes that the SK MIN (SK and MIN supplied by Statiatlcs
unit)
(4 bits) ( I bit)
A and C registers are quite long, Le., that they increase in
length with the number of events encoded. As the value of
variable E increases, so does the length of A and C. In reality,
the A and C registers can be made of fixed precision, and the ADD+/
results shifted left out of the C register. Also, the leading Os of Bit-yerial shift-In of code-strlng blts
Serlal-by-bit FIFO
the A register are not needed, which justifies a fixed-length A f“buffer storage unlt

register. Let A, C,and W denote fixed-length registers (perhaps

no more than 16 bits, for example). - with “Add I ” capablllty

Bit-wrlal \hltt-out of code-string hits

In making the adjustment tofixed-length registers C and A,

(supplied by
Eq. (1) and Eq. (2) are normalized to operate onA: Model unit)
MOUI
( I blt)
W(F) = 2-sK, (to Model unit)

Figure 7 Block diagram of a binary event encoder/decoder system.

W(T) = A - 2-sK
The widths W may have leading Os, and when becoming the
new A we must renormalize. Thus, as theA register develops
leading Os, a normalization left shift restores value 1 to the
most significant bit of the A register. However, A and C must
maintain their relative alignment, so we have a rule: Register
C undergoes a normalization shift whenever register A does.

The basic conceptual input/output view of the algorithm,

both forthe compression and decompression process, is shown
in Figure 7. In this figure we have decomposed the encoding
task into the encoder itself which handles the A and C registers,
and the special arbitrarily long FIFO buffer Q which handles
the code string and the carry-over. Note that the encoder and
decoder in practice are interfaced to the original data via a
statistics unit which is not shown. The statistics unit provides
I I Q,C t \hl Q,C,O
skew numbers SK. t t
N0
DONE’?
The encoder acceptssuccessive events of binary information
from the inputvariable MIN, whose values T or F are encoded
under a coding parameter SK. The code string is shifted out
of the encoder into a FIFO storage unit Q. At the other end
of Q, the decoder receives the code string from FIFO Q. The
EXIT
c
decoder is also supplied the same value of the coding param-
eter SK under which the successive input values MIN were Figure 8 Flowchart for BAC algorithm encoder.
encoded. The decoder, with a knowledge of the code string C
coming from Q at one input port and the successive coding
parameter values at the other inputport SK coming from the
decoder’s statistics unit, produces successive values at output The BAC encoder algorithm with normalization is described
port MOUT. in flowchart form in Figure 8. The figure uses a simple register
transfer notation we define in the obvious way by example.
For the description of the algorithm given here, we assume We assume theexistence of three reasters:Q (the FIFObuffer),
that the FIFO store Q has sufficient capacity for the entire C, and A. Let us assume that values of SK are limited to 1, 2,
code string, and that Q has the capability of propagating an and 3 . Registers C and A are onebit longer than the maximum
incoming carry. Thus, in Fig. 7 the encoder has an output SK, and so are 4 bits. Register C is initialized to 0.000, and
ADD+l which signifies a carry-over from the encoder to the Register A is initialized to 1.000. The basic idea of the algo-
FIFO buffer Q.When ADD+l is active, the FIFO buffer first rithm using normalization is very simple. To encodea T, add
propagates the carry (adds 1 to the lowest-order bit position 2-sK to C and subtract 2-sK from A. Let “,” denote concaten-
of Q) and then performs the shift. ation. In the register transfer notation. 143

IBM J. RES. DEVELOP, VOL. 28 NO. 2 MARCH 1984 GLEN G. L A N C W N . JR.

 Table 4 Example encoding with normalization. Since the register A result is greater than 1.O, the normalization
shift of Eq. ( 5 ) is not needed.
Event MIN SK Q C A Normalization
____
Initial - - - 0.000 1.ooo - Event 3 encodes value F at skew 1. The algorithm for
1 T 3 0 0.010 1.110 Yes encoding an F is Eq. (6). The value F is encoded by shifting
2 T 1 01.0100.110 No the Q,C pair left SK bits and reinitializing A to 1.000. Sum-
3 F 1 00 1.000
1.100 F-shift of SK
4 T I 010 0.000 1.000 Yes marizing Event 3, an F of skew I is a one-bit shift left for Q,C,
so Q is 00 and C is 1.100. Equation (6b) reinitializes A to
1 .ooo.
Q,C e Q,C + 2-sK, (44 Event 4 illustrates a carry-over. Event 4 encodes value T
with a skew of 1. Following Eq. (4a), adding 2” (0.100) to C
A c A - 2-sK. (4b) ( 1.100) results in 10.OOO, where indicates the carry-out from
If the result in A is equal to or greater than I .OOO, we are done the C register. This carrypropagates to Q by activating encoder
with the double recursion. If the result in A is less than 1.OOO, output signal ADD+I, and this carry-over operation converts
a normalization shift is needed. We shift Q and C left as a Q from 00 to 0 1. Meanwhile, for Eq. (4b), 2” subtracts from
pair, and shift left A. Let “shl” denote shift left one bit, and register A leaving 0.100, so the normalization shift of Eq. ( 5 )
“shl’” denote a shift left of two bits, etc. If A is less than I .OOO, is needed. Q now becomes 010. The value of code string is
then 0100000, which is the same result of Table 3, as expected.

Q,C +- shl Q G O , (54 Carry-over control

Arithmetic coding ensuresthat no future value of Ccan exceed
A t shl A,O. (5b)
+
the current value of C A . Consequently, once a carry-over
In the above, “,O” denotes “0-fill” (the vacated positions are has propagated into a given code-string positionin Q, no other
filled with Os). carry-over will reach the same code-stringposition. In the
above sample, the second bit of the code string received a
If the symbol to be encoded is F, Fig. 8 shows that the carry-over. The algorithm ensures that this same bit position
action to perform is relatively simple: (second from the beginning) cannot receive another carry-
over during the succeeding encoding operations. This obser-
Q,C e shlSKQ,C,O, (64
vation leads to a method for controlling the carry called bit-
A t 1.0. (6b) stufing [3]. At the encoder side, if 16 Is in a row are emitted,
the buffer can insert (stuff) a 0. This 0 converts to a I and
We use the same example as in Table 3 redone as shown in blocks the carry from reaching the string of 16 Is. Therefore
Table 4. Columns Q, C, and A show the result of applying the a bit-stuff permits the block with the 16 Is to be transmitted.
MIN and SK values of that step. The first row is the initiali- At the decoder side, if the decoder encounters 16 Is in a row,
zation. the decoder buffer removes and examines the stuffed bit. If
the stuffed bit value is I , the carry is propagated inside the
Event I , with C and A as initialized, encodes value T with decoder.
an SK of 3. The arithmetic result for Eq. (4a) is C = 0.000 + Code-string termination
0.001, and for Eq. (4b) is A = 1.000 - .OOl = 0.1 1 1. Since
When the encoding is done, the C register may be shifted into
0. I I I is less than 1.O, we must apply Eq. ( 5 ) to normalize.
the Q FIFO store.However,after the last eventhas been
Following the normalization shift, Q is now 0, C is 0.010, and
coded, we remain with interval [C,C + A ) . Ifwe know the
A is 1.1 10.
length of the datastring, then we know whenwe have decoded
the last event. Therefore, any code string whose magnitude
Event 2 encodes value T with a skew of 1. We perform the
operations of Eq. (4a), resulting in C of 0. I 10 as follows:
+
lies in [C,C A ) decodes to the original data string. In the
present case, we can simply truncatethe trailing Os. The
0.010 (old C) truncation process leaves “01” as the code string, with the
+-
.I (2-l) convention that the decoder insert as many trailing Os as it
0.1 10 (new C) takes to decode four databits.
Equation (4b) gives 1.O 10 for A:
In the general case, any code string greater than 0100000
1.1 10 (old A) (smallest value in current interval) and strictly less than C +
- .I (-2”) A = 010000 + 0001000 = 010100 suffices. Our shortest
144 1.010 (new A) selection remains 0 I .

GLEN G.LANGDON. JR. IBM J. RES. DEVELOP. VOL. 28 NO. 2 MARCH 1984
 Decoding process START

The decoding part of the BAC algorithm is shown in Figure

9. Consider decoding the example code string of Table 4.We INITIALIZE
C shl' Q,O
demonstrate the decoding process with the aid of Table 5.
Register A is initialized to 1.000 and C is initialized to the
first four bits out of the FIFO buffer Q. Since Q only has 2
Pet SK
bits (OI), we insert Os as needed. C is initialized to 0.100. The
description of each eventthat follows accompanies Fig. 9 and
one line of Table 5.
Event I To decode the first event we need the value of SK,
which is 3. We subtract 2" from C as an intermediate result
called CBUF. CBUF is 0.01 I , which is greater than 0, so the
resulting bit is T. So 2-' is subtracted from A, giving 0.1 1 1
and the contentsof CBUF are transferred into C. Register A
is less than 1.0, so a normalization shift is needed. C and A
arenow0.110and 1.110.
A t shl A
Event 2 Now we obtain the second value of SK, which is I .
Subtracting 2", or 0.100, from C gives a CBUF value of
0.0 IO, which is positive. Therefore the result is T, and 0.010
f
is the new value of C. Subtracting 0.100 from register A gives EXIT
1.O IO, so no normalization is needed.
Figure 9 Flowchart for BAC algorithm decoder.
Event 3 For thethird event, SK is again I , so we again
subtract 0.100 from C (which is now 0.010). The result CBUF
is negative, so the event is F. We do not load C from CBUF,
but simply shift C one position left, and reinitialize A. C is
now 0.100 and A is 1.OOO.

Event 4 The fourth SK is I , and subtracting 0.100 from C

leaves 0.000 for CBUF. The result is not negative, so the event
A = 1.000
is T. To continue the algorithm, we subtract 2" from A,
S u b d l v l w n point
discover that the result 0.100 is less than I , and do a normal- f o r SK = 3
(a)
ization shift of C and A. A is now 1.000 and decoding is
complete.

Note that column A and the Normalization columns of

Table 4 (encoder) and Table 5 (decoder) are identical. The A
register contents always follow the same sequence of values
for the decode phase as for the encodephase.
f-
C = 0 0 0 1 unnornmalized
v
A = I 110 (normallzed)
C =(1.010 dfter norrnallzatmn
Framework for prefixcodes and arithmetic codes (bl
We can apply the code-string-tree representation of the coding
Figure 10 Code-string tree for Event I , Table 4: (a) Initial tree. (b)
operations to arithmeticcodes. However, unlike prefix codes, Following encoding of Event 1.
in arithmetic coding the encoding of a given symbolmay
result in a code spacedescribed by continuations of more than
one leaf of the tree. We illustrate the point by showing the
example of Event 1 of Table 4 in the form of a code-string Table 5 Example decoding.
tree of Figure 10.
Event SK C (ajier) A (aJier) CBUF MOUT Normalization
-~
The smallest subinterval at the current depthis a single leaf. Initial - 0.100 1.000 - - -
With a maximum skew SK of 2-3, we identify value ,001 with I 3 0.110 1.110 0.011 T Yes
a single leaf at the current depth. With a maximum SK of 3, 2 I 0.010 1.010 0.010 T No
3 I 0.100 1.000 -1.110 F F-shiftofSK
the value of A can range from 1.OOO to 1.1 IO. At the same 4 I 1.000
0.000 0.000 T Yes
current depth where 2" is one leaf, the subset of code-string 145

IBM J . RES. DEVELOP. VOL. 28 NO. 2 MARCH 1984 GLEN G. L A N G W N . JR.

 generalization of Golomb's code [4]. However, an interesting
relationship exists, which the readers are invited to discover
for themselves. We suggest that a Golomb code be selected,
that a short run be encoded, and that thereader construct the
corresponding code-string tree.
C=O.OIl unnormalized
. A = 1.010
C = O . 110 after normalization
Subdivision polnt
3. The general case: Multisymbol data alphabets
(a) As noted from the exampleof Table 2, arithmetic coding
applies to n-symbol alphabets as well. Here we use notation A
and C in the arbitrarily long register sense of Table 3, and
A(s) and C(s) are therespective values following the encoding
of data string s. We describe some encoding equations for
multisymbol alphabets.

Encoding
Let the n-symbol alphabet, whatever it may be, have an
A = I 000 ordering. Our interest is in the symbol positionof the ordering:
(b) 1, 2, . . ., k, . . ., n. Let relative frequencies p l ,p2, . . ., pn,and
interval width A ( s ) be given. The interval is subdivided ac-
Figure 11 Code-string tree for Event 3, Table 4: (a) Following Event cording to relative frequencies, so the respective widths are
2. (b) Following Event 3.
p I X A(s),pz X A(s), . . ., pn X A(s). Recall the calculation of
cumulative probabilities P :

tree leaves which describe the code space is from eight leaves Pk = pJ for k 2 2, and PI = 0.
Jck
SK of 2 correspondsto two
to 14 leaves. Also at this depth, an
leaves and an SK of 1 corresponds to four leaves. The initial For encoding the eventwhose order is k, following the encod-
code-string tree has eight leaves (A = 1.000) and a depth of ing ofpriorstring s, the new subinterval [C(s,k),C(s,k) +
three. See Fig. 10(a) for the subdivision point for Event I of A(s,k)) is described with the following two equations of the
Table 4. Event I is T, so we keep the right subset of seven familiar arithmetic coding double recursion:
leaves. For subsequentencoding, the codestring will be a C(s,k)= C(s) + D(s,k),where D(s,k) = A ( s ) x Pk, (7)
continuation of 001, 0 1, or 1, and we clearly are not dealing
with a prefix code. A(s,k)= A ( s ) X P k . (8)

With only seven leaves at depththree, we increase the depth Decoding

by one to four, so that we now have 14 leaves at the new Let I C(s) I denote the result of the termination operation on
depth. Thisprocess of increasing the depthcorresponds to the C(s),where it is understood that the decoder inserts trailing
normalization shifting done in Table 4. The result is shown Os as needed. The decoding operation, also performed recur-
in Fig. 10(b). sively per symbol, may be accomplished by subtracting out of
I C(s) I the previously decoded summands. Forexample, let so
Figure 11 shows Event 3 of the example. Figure I I (a) shows be the prefix of data string s which has already been decoded.
the subdivision point for the skew of 1 for Event 3. An SK of We are now left with quantity I C(s)I - C(so). The decoder
1 corresponds to a subinterval of four leaves, which are the must find symbol i such that
leftmost leaves of the current interval. The event value is F,
D(so,k)5 I C(S)1 - C(X')< D(so,k+ I),
so inthis case we retain the left subinterval with the four
leaves. Figure 1 I(b) shows the result after encoding Event 3. where k < n. We decode symbol n if D(so,n)5 I C(s) I - C(so).
Since only four leaves were taken, we increase the tree depth
one level, again analogous to the normalization operation in Precision problem
the example of Table 4. Now the current code space which Equations (7) and (8) describe Elias' code, which has a preci-
we subdivide for Event4 consists ofthe continuationsof code sion problem. If A ( s ) has 8 bits of precision, and P k has 8 bits
strings: 01 1 and 100. of precision, then A(s,k) requires 16 bits of precision. If we
keep the 16-bit result, A needs 24 bits of precision for the next
We close this section with a suggestion. The path by which recursion, and so on. In practice, we wish A(s,k)to have the
we discovered the particular binary arithmetic code described same precision as A(s),and use fixed-precision registers A and
146 here [2] was asimplificationof the codein [3] and was not a C for the arithmetic as we do in theBAC examples of Tables

GLEN G. LANGDON. JR IBM J. RES, DEVELOP. VOL. 28 NO. 2 MARCH 1984

4 and 5. Pasco [ 5 ] , who is responsible for the first FIFO X ( s ) is a rational fraction of
an integer denominator M. In
arithmetic code, solved the precision problem by truncating fact, the lengthof the codestringfor s, denoted L(s), is
the product of Eq. (8) to the sameprecision as A(s).In terms +
Y(s) X@). Here, Y ( s )corresponds to E(s) of the example of
of the leaves at the current depth of the code-stringtree, Table 3. [When the code string is terminated following the
truncation in Eq. (8)allows gaps to form in thecode space. A encoding of the last symbol, the code-string length of C(s)is
gap is a leaf of the current interval which is not assigned to within a few bits of L(s).]For L-based arithmetic codes, D(s,i)
any of the symbols. For such unassigned leaves, no datastring is determined as the product
maps to a continuation ofsuch leaves, and there is some
D(s,k) = D(X(s),k)x 2TY‘”), (12)
wasted code space. Martin [6] and Jones [7], independently
of Pasco and each other, discovered practical versions of Eqs. so that
(7) and (8) which partition the code space and have no gaps.
C(s,k)= C(s)+ D(s,k).
They do this by ensuring that
Since there are M distinct values of X($), i.e., 0, 1/M, 2/
C(s,k + 1) = C(s,k)+ A(s,k).
M , . . ., ( M - l ) / M , and n symbols, we precalculate an M-by-
Decodability n table of values D(X,k). [Actually, for symbol k = 1, the
Codes are not uniquely decodable if two data strings map value of D is 0, SO only M-by-(n - 1) values need be stored.]
into the samecode space. In arithmetic codes, if the subdivi- In Eq. (12), multiplication by 2-‘(”) is simply a shift.
sion of the interval yields an overlap, more than one data
string can map to the overlappedsubinterval.Overlap oc- Corresponding to the relative frequency estimates, pkrare
curs when, in subdividing interval A(s), the sum of the sub- instead length approximations, 4 = -log P k . These lengths
interval widths exceeds A(s) itself. Thus, some continuations must satisfy the generalized Kraft inequality [8].Following
of data string s may map to thesubinterval whose least point encoding of symbol k, the internal variables are updated:
is C(s + I). We define string s + 1 to be the data string of the
L(s,k)= Y(s)+ X ( s ) + 4 ,
same length as s which is next in the lexical ordering. If s is
already the last string in the ordering, i.e., string “n n . . .n,” where again L(s)is broken into integer part Y(s,k)and fraction
then C(n n . . . n) is 1.OOO. The general decodability criterion part X(s,k).The decodability criterion of Eq. ( 1 1) is also the
is thus decodability criterion for L-based codes if A(s,k) is as defined
in Eq. (13).
C(s,n)+ A(s,n)c C(s + I), (9)
where s,n denotes symbol n concatenatedto string s. For For k < n define
arithmetic codes which do not explicitly use value A(s,n), it
A(s,k)= B(s,k + 1) - B(s,k),
may be calculated as C(s n n n n, . . .) - C(s,n).In cases where
Eq. (9) is violated, the interval of overlap is and for n,

[C(s + I),C(s,n)+ A(s,n)). A(s,n) = B(s,n,n) + B(s,n,n,n) + . . .. (13)

Similarly, a gap is formed between the mappings ofdata string

Applications
s a n d s + 1 if
Langdon and Rissanen [ 3 ] applieda FIFO P-based binary
C(S+ I ) - C(S)
> A(s,l) + . . . + A(s,n). (10) arithmetic code to the encoding of black and white images,
using as a contextthe value of neighboring pels already
A gap does not affect decodability; it simply means that the
encoded. This work introducedthe notion ofthe binary coding
code space is not fully utilized.
parameter called skew. A binary codeis particularly useful for
P-based arithmetic codes black and white images because the contexts employed for
For P-based arithmetic codes, the code space is represented as successive pels may have different relative frequencies. Tra-
a number A which is subdivided by a multiplication in pro- ditional run-length codes such as Golomb’s [4] are only de-
portion to therelative frequencies of the symbols. signed for aparticular relative frequency for the repeating
symbol. In [ 3 ] , amethod to dynamically adapttothe pel
The decodability criterion for P-based codes is given by statistics is described.Particularlysimple adaptation tech-
A(s) 2 A ( $ , ] )+ A(s,2)+ . . . + A(s,n). (1 1) niques exist for determining the skew number.

If this equation is met with equality forall s, then thealgorithm

Arithmetic coding can also be applied to file compression
leaves no gaps.
[5-7,9, I O ] . In [9], the first-order contexts are also determined
L-based arithmetic codes adaptively. A linearized binary tree can be used to store the
The L-based arithmetic codes represent the width of the code skew numbers required to encode the decomposed 8-bit byte
space A(s) as a value 2-[y(s)+x(s)1,
where Y(s) is an integer and as eight encodings, each of a binary event. 147

IBM J. RES. DEVELOP. VOL. 28 NO. 2 MARCH 1984 GLEN G.LANGDON. JR

 Arithmetic codes have been employedto design codes for a reader, it is due toJoan’s graciousand patient encouragement.
constrained channel. A constrained channel has an alphabet, I thankGerry Goertzelfor his insights in explaining the
e.g. (0,1], where not allstrings in (0,1]* are allowed. The operation of arithmetic coding. I have also benefited from the
Manchester code, where clocking and data information are encouragement of Janet Kelly, Nigel Martin, Stephen Todd,
recorded together, is an example of a constrained channel Ron Arps, and Murali Varanasi.
code used in magneticrecordingchannels. Shannon [ I I ]
studied the constrained channel, defined the constraints in Appendix 1: Origins of arithmetic coding
terms of allowed transitions in a state table, and determined The first step toward arithmetic coding was taken by Shannon
the probabilities for the allowed transitions which are needed [ 1 I], who observed in a 1948 paper that messages N symbols
in order to map the maximum amount of information to a long could be encoded by first sorting the messages in order
set of allowed strings. Guazzo [ 121 showed that mapping data of their probabilities and then sending the cumulative proba-
strings to channel strings is analogous to the decompression bility of the preceding messages in the ordering. The code
operation, and recovering the data string from the channel string was a binary fraction and was decoded by magnitude
string corresponds to thecompression operation. Martin eta]. comparison. The next step was taken by Peter Elias in an
[ I ] showed thatthedual of decodability forcompression unpublished result; Abramson [ 161 described Elias’ improve-
coding is representability for constrained channel coding. To ment in 1963 in a note at the endof a chapter. Elias observed
be representable, the encoding operation can have overlaps that Shannon’s scheme worked without sorting the messages,
but cannot have a gap. Some interesting L-based codes for and that the cumulative probability of a message of N symbols
constrained channels arederived in [ 131. could be recursively calculated from individual symbol prob-
abilities and the cumulative probability of the message of N
Guazzo [ 121, in addition to suggesting the application to - 1 symbols. Elias’ code was studied by Jelinek [ 171. The
constrained-channel codes, contributed a code which subdi- codes of Shannon and Elias suffered from a serious problem:
vides the codespaceaccording tothe given probabilities. As the message increased in length the arithmetic involved
However, the subdivision method is quite crude compared to required increasing precision. By using fixed-width arithmetic
Pasco’s [5]. units forthese codes, thetimetoencode eachsymbol is
increased linearly with the length of the code string.
4. Comments
Arithmetic codescan achieve compressionas close to theideal Meanwhile, another approach to coding was having a sim-
as desired, for given statistics. Inaddition,the P-based ilar problem with precision. In 1972, Schalkwijk [ 181 studied
FIFO arithmeticcodes which accept statistics directly facilitate coding fromthestandpoint of providing an index tothe
dynamic adaptation to the statistics in one pass of the data encoded string within a set of, possible strings. As symbols
[3]. A good binary code is important, as Shannon and others were added to the string, the index increased in size. This is a
havenoted, because other alphabets can be converted to last-in-jirst-out (LIFO) code, because the last symbol encoded
binary form by decomposition. was the first symbol decoded. Cover[ 191 made improvements
to this scheme,which is nowcalled enumerative coding. These
Of ovemding importance tocompression now is the mod- codes suffered from the same precision problem.
eling of the data. Rissanen and Langdon [ 141 have studied a
framework for the encoding of data strings and have assigned
Both Shannon’s code and the Schalkwijk-Cover code can
a cost to a model based on the coding parameters required.
be viewed as a mapping of strings to a number, forming two
Different modeling approachesmay be compared. They
branches of pre-arithmetic codes, called FIFO (@st-in-jirst-
showed that blocking to form larger alphabets results in the
out) and LIFO. Both branches use a double recursion, and
same model entropy at the samecost in coding parameters as
bothhave a precision problem.Rissanen [8] alleviated the
a symbolwise approach. In general, the compression system
precision problem by suitable approximations in designing a
designer seeks ways to give up a small percentage of the ideal
LIFO arithmetic code. Code strings of any length could be
compressionin order to simplify the implementation. The
generated with a fixed calculation time per data symbol using
existence of an efficient coding technique now places the
fixed-precision arithmetic.
emphasis on efficient context selection and parameter-reduc-
tion techniques [ 141.
Pasco [ 5 ] discovered a FIFOarithmetic code, discussed
earlier, which controlled the precision problem by essentially
Acknowledgments the same ideaproposed by Rissanen. In Pasco’s work, the
Most of the author’s work in this field was done jointly with code stringwas kept in computer memory until last the symbol
J. Rissanen, and thisdebt is obvious.I also owe a debt to Joan was encoded. This strategy allowed a carry-over to be propa-
Mitchell, who has made several contributions to arithmetic gated over a long carry chain. Pasco [5] also conjectured on
148 coding [ 151. If this paper is more accessible tothe general the family of arithmetic codes based on their mechanization.

GLEN G. LANGDON. JR. IBM J. RES. DEVELOP. VOL. 28 NO. 2 MARCH 1984
In Rissanen [8] andPasco [ 5 ] , theoriginal (given, or pre- 10. Frank Rubin, “Arithmetic Stream Coding UsingFixed Precision
sumed) symbol probabilities were used. (In practice, weuse Registers,” IEEE Trans. Info. Theory lT-25,672-675 (November
1979).
estimates of the relative frequencies. However, the notion of 11. C. Shannon, “A Mathematical Theory of Communication,” Bell
an imaginary “source” emitting symbols according to given Syst. Tech. J. 27, 379-423 (July 1948).
probabilities is commonly found in the coding literature.) In 12. Mauro Guazzo,“A General Minimum-RedundancySource-Cod-
ing Algorithm,” IEEE Trans. Info. Theory IT-26, 15-25 (January
[20] and [3], Rissanen and Langdon introduced the notion of 1980).
coding parameters “based” on the symbol probabilities. The 13. StephenJ. P. Todd,GlenG.Langdon, Jr., and G. Nigel N.
uncoupling of the coding parameters from the symbol proba- Martin, “A General Fixed Rate Arithmetic Coding Method for
ConstrainedChannels,” IBM J. Res. Develop. 27,107-1 15
bilities simplifies theimplementation of the code at very little (1983).
compression loss, and gives the code designer some tradeoff 14. J. J. Rissanen and G. Langdon, “Universal Modeling and Cod-
possibilities. In [20] it was stated that there were ways
to block ing,” IEEE Trans. Info. Theory IT-27, 12-23 (January 1981).
15. D. Anastassiou, M. K. Brown, H. C. Jones, J. L. Mitchell, W. B.
the carry-over, and in [3] bit-stuffing was presented.In [ 101 F. Pennebaker, andK. S. Pennington, “Series/l-Based Videoconfer-
Rubin also improved Pasco’s code by preventing carry-overs. encing System,”IBM Syst. J. 22,97-I 10 (1983).
The result was called a “stream” code. Jones [7] and Martin 16. N. Abramson, Information Theory and Coding, McGraw-Hill
Book Co., Inc., New York, 1963.
[6] have independently discovered P-based FIFO arithmetic 17. F. Jelinek, Probabilistic Information Theory, McGraw-Hill Book
codes. Co., Inc., New York, 1968.
18. J. Schalkwijk, “An Algorithm for Source Coding,” IEEE Trans.
Info. Theory IT-18, 395 (1972).
RissanenandLangdon [20] successfullygeneralizedand 19. T. M. Cover, “Enumerative Source Coding,” IEEE Trans. Info.
characterized the family of arithmetic codes through the no- Theory IT-19, 73 (1973).
tion of thedecodabilitycriterionwhichapplies to allsuch 20. J. J. Rissanen and G. G. Langdon, Jr., “Arithmetic Coding,” IBM
J. Res. Develop. 23, 149-162 (1979).
codes, be they LIFO or FIFO, L-based or P-based. The arith- 21. E. N. Gilbert and E. F. Moore, “Variable-Length Binary Encod-
metic coding family is seen to be a practical generalization of ings,” Bell Syst. Tech. J. 38, 933 (1959).
many pre-arithmeticcoding algorithms, including Elias’code, 22. J. Rissanen, “Arithmetic Coding as Number Representations,”
Acta Polyt. Scandinavica Math. 34,44-5 I (December 1979).
Schalkwijk [ 181, andCover [ 191. GilbertandMoore [21]
devised the prefix coding approach used in Table 1. In [22],
Rissanen presents an interesting viewof an arithmeticcode as Received June 29, 1983; revised October 8, 1983
a number-representation system, and shows that Elias’ code
and enumerative codes are duals.
Glen G. Langdon, Jr. IBM Research Division, 5600 Cottle
Road, San Jose, California 95193.Dr.Langdon received the B.S.
References from Washington State University, Pullman, in 1957, the M.S. from
I . G. Nigel N.Martin,Glen G. Langdon,Jr.,andStephenJ. P. the University of Pittsburgh, Pennsylvania, in 1963, and the Ph.D.
Todd,“Arithmetic Codes for ConstrainedChannels,” IBM J. from Syracuse University, New York, in 1968, all in electrical engi-
Res. Develop. 27, 94-106 (March 1983). neering. He worked for Westinghouse on instrumentation and data
2. G. G. Langdon, Jr. and J. Rissanen, “A Simple General Binary logging from 196 I to 1962 and was an application programmer for
Source Code,” IEEE Trans. Info. Theory IT-28, 800-803 (Sep the PRODAC computer forprocess control for most of1963. In 1963
tember 1982). he joined IBM at the Endicott, New York, development laboratory,
3. Glen G. Langdon,Jr.andJormaRissanen,“Compression of where he did logic design on small computers. In 1965 he received an
Black-White Images with Arithmetic Coding,’’ IEEE Trans. Com- IBM Resident Study Fellowship. On his return from Syracuse Uni-
mun. COM-29,858-867 (June 1981). versity, he was involved in future system architectures and storage
4. S. W. Golomb, “Run-Length Encoding,” IEEE Trans. Info. The- subsystem design. During 1971, 1972, andpart of 1973, he was a
ory IT-12, 399-401 (July 1966). Visiting Professor at the University of Sao Paulo, Brazil, where he
5. R. Pasco, “Source Coding Algorithms for Fast Data Compres- developed graduate courses on computer design, design automation,
sion,” Ph.D. Thesis, Department of Electrical Engineering, Stan- microprogramming,operating systems, andMOS technology. The
ford University, CA, 1976. first Brazilian computer, called Patinho Feio (Ugly Duckling), was
6. G. N. N. Martin, “Range Encoding: an Algorithm for Removing developed by the students at the University of Sao Paulo during his
Redundancy from a Digitized Message,’’ presented at the Video stay. He is author of Logic Design: A Revien of Theory and Practice,
andDataRecordingConference,Southampton,England,July an ACM monograph, and coauthor of the Brazilian text Project0 de
1979. Sistemas Digitais; he has recently published Computer Design. He
7. C. B. Jones, “An Efficient Coding System for Long Source Se- joined the IBM Research laboratory in 1974 to work on distributed
quences,” IEEE Trans. Info. TheoryIT-27,280-291 (May 1981). systems and later on stand-alone color graphicsystems. He has taught
8. J.J.Rissanen,“GeneralizedKraftInequalityandArithmetic graduate courses on logic and computer design at the University of
Coding,” IBMJ. Res. Develop. 20, 198-203 (1976). Santa Clara, California.He is currently working in data compression.
9. Glen G. Langdon, Jr. and Jorma J. Rissanen, “A Double Adaptive Dr. Langdon received an IBM Outstanding Innovation Award for his
File Compression Algorithm,” IEEE Trans. Commun. COM-31, contributions to arithmetic coding compression techniques.He holds
12.53-1255 (1983). eight patents.

149

IBM J . RES I3EVELOP. VOL. 28 NO. 2 MARCH I 984 GLEN G LANGDON. JR.