LET'S GET TECHNICAL (Article from October 2004) Shrinking Bits: A Second Look at Digital Data Compression James

L. Antonakos Last time we examined the applications for lossless and lossy data compression methods. In this second look at digital data compression, we will take a look inside these different compression techniques: Run Length Encoding Dictionary-Based Compression Huffman Coding Shannon-Fano Coding Quantization

All of these techniques provide lossless compression except for the Quantization, which throws away information and achieve higher compression ratios. Let us begin with Run Length Encoding. Run-Length Encoding (RLE) is one of the most simplest compression schemes available. In this technique, a single data value and a repeat count replace consecutive data values that have the same value. Figure 1 shows a simple example. Here, 20 bytes of input data are compressed into 12 bytes of output data. The more the data values stay the same, the greater the compression. For example, if 255 bytes of data all contain the same value, the RLE data will consist of only two bytes (repeat count and data value).

Figure 1: Using run-length encoding to compress data Dictionary-based compression involves building a dictionary of the words (or phrases) used in the text to be compressed. Pointers to the words within the dictionary represent the words from the input text. Frequently used words are only stored once in the dictionary, which is where the compression comes in. Figure 2 illustrates this compression process.

Figure 2: Operation of the word-based dictionary compression process Let us work through an example. Consider the following block of text (which is contained within the input file): 2

The wheels on the bus go round and round round and round round and round. The wheels on the bus go round and round all day long. This text contains 138 bytes of data (uncompressed) when saved as an ordinary text file. Do not forget that the newline characters (carriage return, line feed) at the end of each line must also be counted, as well as the spaces between words. The word dictionary created for this input data contains the entries shown in Table 1. Word Number 1 2 3 4 5 6 7 8 9 10 11 12 13 Word The wheels on the bus go<cr> round and round<cr> round. all day long. Total Length Length 4 7 3 4 4 3 6 4 6 7 4 4 6 62

Table 1: Word dictionary created from input text file The pointers to the words in the dictionary will be saved as 16-bit (two byte) integers. This allows for 65,535 different words in the dictionary. The pointer stream for the input text looks like this: 1 7 7 7 1 7 11 0 2 8 8 8 2 8 12 3 9 9 10 3 9 13 4 5 6

The 0 pointer at the end indicates the end of the pointer stream. This gives a total of 28 pointers, which require 56 bytes of storage in the output file. Together with the 62 bytes of dictionary text, the output file contains a total of 118 bytes. This is not much of a savings compared to the original 138 bytes of uncompressed data. If, however, the words were longer or occurred more frequently, 3

better compression would result. Let us see if this is true by extending words into phrases. Table 2 lists the phrases found in the input file. Phrase Number 1 2 3 4 Phrase The wheels on the bus go round and round round and round. all day long. Total Length Length 25 16 17 14 72

Table 2: Phrase dictionary created from input text The pointer stream for the phrases becomes: 1 2 2 3 1 2 4 0 Now, we only need to store eight phrase pointers, for a total of 16 bytes. Together with the 72 bytes of phrase dictionary we have an output file containing 88 bytes of compressed data, a much larger savings than the word-based method. Both RLE and dictionary compression do their work on-the-fly. Other compression techniques look at the entire block of data before beginning their work. These compression techniques fall into the Statistical category of compression methods. The first technique in this category is Huffman coding. In this technique, we build unique binary strings to represent the different data items we encounter. The binary strings typically require fewer bits to store than the original data item. Huffman coding begins with information that describes the distribution of different data items within the entire block of data. For example, suppose we have a text file containing 250,000 characters, all of which are either A, B, C, or D, with the percentages shown in Table 3. Character Percentage A 50% B 10% C 15% D 25%

Table 3: Distribution of data items within input file Huffman coding begins by finding the two smallest percentage items (the B and C characters) and combining them into a simple tree structure. The combined percentage (10% plus 15% equals 25%) is now placed back into the list of percentages and the process is repeated until you get to 100%. The data items are placed into the structure so that the lower percentage item is always on the left. Figure 3 shows the tree structure generated using the Huffman technique.

Figure 3: Tree structure containing unique binary strings for each data item. By traversing the tree, we can determine the unique binary strings associated with each data item. Table 4 shows the results of the traversal. Data Item A B C D Percentage 50% 10% 15% 25% Bit String 0 110 111 10

Table 4: Huffman coding strings for the four data items Notice that the data items with the largest percentages have the smallest bit strings. This is the beauty of Huffman coding. Knowing the lengths of each bit string, we can easily determine the average number of bits per character required in the compressed file. Table 5 shows the calculations. Data Item A B C D Bit String 0 110 111 10 Bit Length Percentage 1 3 3 2 50% 10% 15% 25% Total Bits Required 0.5 0.3 0.45 0.5 1.75

Table 5: Calculations to determine average number of bits per compressed character

Multiplying each percentage by its associated bit string length and adding them up gives a total of 1.75 bits per compressed character. In the original data file, each character required eight bits of storage for a total of 2,000,000 bits. Now only 437,500 bits are needed (250,000 characters times 1.75 bits/character), plus a few bits to store the unique strings table and associated data items. You can experiment with Huffman coding through a simple MSDOS program called HUFF, available for download at http://www.sunybroome.edu/~antonakos_j/nutsvolts/huff.exe A sample execution of HUFF for the previous example is shown in Figure 4.

Figure 4: Sample execution of HUFF program 6

The percentages on the MSDOS command line represent, in order, the As, Bs, Cs, and Ds and must add up to 100%. The Shannon-Fano coding technique also uses the list of percentages to determine unique bit strings for the individual data items. Instead of building a tree structure, the Shannon-Fano technique simply breaks down the data items into different groups of items, assigning a bit value to each group. First, the items are arranged in descending order of percentage, as shown in Table 6. Data Item A D C B Percentage 50% 25% 15% 10%

Table 6: Percentages sorted into descending order Next, divide the items into two groups so that each group has roughly the same percentage as the other. One group gets a 0 bit assigned to it and the other group gets a 1 bit. Keep subdividing the groups until there are no more groups to split. Figure 5 illustrates this process.

(a) (b) Figure 5: Partitioning the data items in Shannon-Fano coding (a) Finding the first two groups (A and DCB) (b) Splitting the DCB group (into D and CB) (c) Splitting the CB group

(c)

The unique bits strings for each data item are easily read off Figure 5(c). Note the similarities with the strings from the Huffman coding example. Are the results the same? If not, are they acceptable? The answers are Not exactly and sure they are! Last, we come to our only lossless technique, buried within the compression algorithm for JPG images, and indicated in the flowchart shown in Figure 6.

Figure 6: Steps involved in compressing one 8-by-8 block of pixels in a JPG image The compression in a JPG comes from the combination of a Quantizing process followed by RLE compression. An algorithm called the Discrete Cosine Transform (DCT) is used on an 8-by-8 block of pixels from the original image, converting the 64 data values in the block to another set of 8-by-8 DCT values. These new values do not represent pixel colors or intensities any longer. Instead, they represent frequency information caused by the interaction of the pixels. If a reverse DCT is used on the converted data you would get the original pixels back. Instead, the Quantizing process divides all the DCT values by an integer, throwing away the remainders. For example, the following string of data is quantized by dividing all values by 10 and ignoring the remainders: Input: Output: 212 21 186 18 112 11 67 6 36 3 18 1 11 1 4 0

Now, when the quantized data is un-quantized (multiplied by 10), we get: 8

Input: Output:

21 210

18 180

11 110

6 60

3 30

1 10

1 10

0 0

Let us compare the original eight data values with their un-quantized values: Original: 212 Un-quantized: 210 186 180 112 110 67 60 36 30 18 10 11 10 4 0

They are all different. Lossy compression does not give us our original data back. But in the case of the JPG image, this does not matter. The un-quantized values will be passed through the reverse DCT process, giving an 8-by-8 block of pixels that are close to the original block of pixels but slightly different. What, only 30 shades of blue instead of 243? Our eye is not good enough to notice subtle changes in color, which is why we can get away with lossy compression (via quantization) in the JPG image. Plus, best of all, by throwing away the remainders, the quantized data compresses better. The nature of the DCT is to create values similar to those shown in the example. However, the DCT values get smaller in each new row of the 8-by-8 matrix, which leads to many 0s and other small integers clustering near the bottom right corner of the matrix. By using a Zig-Zag technique to read the quantized values out one diagonal at a time, we create a 64 element string of quantized values with many duplicated values grouped together. RLE compression then compacts the string by eliminating the duplicates. The process shown in Figure 6 must be repested for every 8-by-8 block of pixels in the image. An image having a resolution of 640 by 480 would contain 4800 blocks of pixels. The DCT process alone would require over 2.4 million multiplications for all pixel blocks. Just seeing a JPG image appear in a browser is a feat of mathematical engineering. There are many other compression methods, some of which are listed in Table 7. Compression Technique Lempel-Ziv-Welch (LZW) Adaptive Huffman Delta Modulation Category and Type Lossless, dictionary Lossless, statistical Lossy via quantization Application TIF image files Large files Speech compression

Table 7: Additional compression techniques Some techniques are easily performed in hardware while others are easily applied using hardware. Even inexpensive digital cameras have hardware to compress the image data. Search the web for additional compression information and techniques, and be prepared to compress the results you get from the volume of information out there.

James Antonakos is a Professor in the Departments of Electrical Engineering Technology and Computer Studies at Broome Community College, with over 28 years of experience designing digital and analog circuitry and developing software. He is also the author of numerous textbooks on microprocessors, programming, and microcomputer systems. You may reach him at antonakos_j@sunybroome.edu or visit his web site at http://www.sunybroome.edu/~antonakos_j.

10