Browse Books

Reviewer: Vladik Ya. Kreinovich

Before Shannon, the word “information” was often used to express vague humanistic notions. Shannon was the first to give this notion a precise and intuitively clear definition: crudely speaking, information is the average number of binary (yes-no) questions that we have to ask in order to make our knowledge complete. For example, if we know only that an experiment can have <__?__Pub Fmt italic>n<__?__Pub Fmt /italic> possible outputs, then we must ask at least log 2 n binary questions in order to determine the actual output. So we need at least log 2 n bits of computer memory to store our knowledge about the actual output (we can store a binary representation of its number). Therefore we say that when someone tells us the actual result we gain log 2 n bits of information. The situation is somewhat different when in addition we know a priori the probabilities p 1 ,&ldots;,p n of possible outputs. If we have only one experiment, all <__?__Pub Fmt italic>n<__?__Pub Fmt /italic> outputs are possible, so we still need log 2 n binary questions to determine which of <__?__Pub Fmt italic>n<__?__Pub Fmt /italic> results occurred. If we have <__?__Pub Fmt italic>N<__?__Pub Fmt /italic> equivalent situations (with the same probabilities), however, not all n N combinations of <__?__Pub Fmt italic>N<__?__Pub Fmt /italic> outputs are possible: combinations are limited by the demand that the answer should be <__?__Pub Fmt italic>i<__?__Pub Fmt /italic> in approximately p i cases. Therefore the number <__?__Pub Fmt italic>Q<__?__Pub Fmt /italic>(<__?__Pub Fmt italic>N<__?__Pub Fmt /italic>) of binary questions that we need to ask in order to determine the outputs of all the experiments is generally smaller than log 2 n N =N log 2 n , so the average number of questions <__?__Pub Fmt italic>Q<__?__Pub Fmt /italic>(<__?__Pub Fmt italic>N<__?__Pub Fmt /italic>)/<__?__Pub Fmt italic>N<__?__Pub Fmt /italic> is smaller than log 2 n . Shannon has shown that when <__?__Pub Fmt italic>N<__?__Pub Fmt /italic> increases, this average number of questions tends to the value - p i log 2 p i (used in statistical physics under the name of entropy). This fundamental result gives an explicit formula for the amount of information when we have a sequence of independent identical events. In real life these events can be correlated, and their probabilities can change with time. How to compute the information then is a question for information theory. The answers are often related to mathematical entropy theory (initially developed for statistical physics). A related question is, If we have correlated random processes <__?__Pub Fmt italic>X<__?__Pub Fmt /italic> and <__?__Pub Fmt italic>Y<__?__Pub Fmt /italic>, and we know the results of <__?__Pub Fmt italic>Y<__?__Pub Fmt /italic>, how many binary questions do we have to ask (on average) in order to determine the results of <__?__Pub Fmt italic>X<__?__Pub Fmt /italic>__?__ (Because of correlation, we can often ask fewer questions than when we do not know the results of <__?__Pub Fmt italic>Y<__?__Pub Fmt /italic>.) Another class of questions is related to information transfer. If we have a channel whose output is distorted by noise, and the probabilities of different distortions are known, then the output contains less information than the input. Since the distortion is of some specific type, we can often decrease the information loss by properly encoding the input signal and correspondingly decoding the output signal (sometimes this is achieved at the expense of slowing down the information transfer, such as when we simply repeat every message twice). Natural questions are What minimal loss of information can we achieve for a given channel by using a proper encoding__?__ and What encoding should we use__?__ These questions also turn out to be closely connected to entropy theory. Gray's book gives a survey of the known results and provides complete proofs for most of them. The book is not easy reading: it is an advanced mathematical text aimed at those who feel comfortable with mathematical papers. For these readers, the book is a gift: it contains everything that was previously spread over papers, preprints, and conference proceedings. The prerequisites for this book are standard results on probability, random processes, and ergodic theory that you can also find under one cover in a previous book by the author [1]. From the applications viewpoint, one of the most promising tendencies in current information theory is the analysis of so-called robust codes (see Section 12.5).<__?__Pub Caret> Traditional information theory is based on an assumption that we know the precise probabilities of all possible distortions, while normally we have only estimates for them. So when we apply traditional methods to these estimates and get an optimal coding, the resulting behavior may be far from optimal in the real situation, where the probabilities are somewhat different. Methods that work well for all possible probabilities within a given precision range are called robust. From the computer science viewpoint certain areas are missing. First, the results presented in this book show only what can be achieved in principle by coding; coding algorithms are rarely given. Algorithms are promised, however, in the author's forthcoming book [2]. Second, Gray makes no mention of the Kolmogorov-Solomonoff algorithmic approach to probability. Conventional probability theory predicts the probabilities of different output sequences, but it does not explain what it means that a given sequence of experimental results is “random,” so it does not explain when a sequence of observations is consistent with an assumption about probabilities. Algorithmic theory describes the notion of a random sequence (or a random process) in algorithmic terms. This approach has been successfully applied to information theory. The third omission (related to robust methods) includes attempts to generalize information theory to the case when some knowledge is not statistical, but subjective (as in fuzzy theory). These additions would demand an increase in the book's size, but I would welcome some references at least. This book is not easy to read, with condensed proofs and no exercises. For a mathematically minded computer scientist, however, this challenge is rewarding.