Probability Concepts and Theory for Engineers

This book is dedicated to students young and old whose thinking will help shape the future.

Preface

This book had its earliest beginnings as a gradually expanding set of supplementary class notes while I was in the Department of Electrical Engineering (later renamed Department of Electrical and Computer Engineering, and now Department of Electrical Engineering and Computer Science) at Syracuse University. Our graduate course in ‘Probabilistic Methods’ was one of the courses I taught quite a few times—in the day and evening programs on campus as well as at the University's off-campus Graduate Centers and via satellite transmission.

Early on I found that existing textbooks, while providing a good coverage of probability mathematics, seemed to lack a consistent and systematic approach for applying the mathematics to problems in the ‘real world.’ My notes therefore focused initially on that aspect. I began to understand that mathematics cannot be ‘applied to the real world.’ One can only apply the mathematics to one's thoughts about the external world. This makes it important that these thoughts are appropriately structured—that a real-world problem gets conceptualized in a manner that allows the correct and consistent application of mathematical principles. This conceptualizing is of little concern in most applications of mathematics since it occurs rather automatically. But the application of Probability Theory calls for much more attention to be devoted to formulating a suitable conceptual model.

Each time I taught the course I tried to improve and add to my notes. After my retirement from the Department I happened to look at these notes again, and they struck me as sufficiently interesting to make it worthwhile to expand and rework them into a textbook. Now, many years later, after incorporation of much more material and a great deal of editing and revising, as well as the creation of a large number of problems and review exercises, here is that book.

Naturally, many individuals and sources have helped me to move forward with this project and to bring it to completion. First, I am indebted to Profs. C. Goffman, H. Teicher, and M. Golomb, among others, in the Department of Mathematics at Purdue University, for helping me strengthen my mathematical thinking. Of course, I derived inspiration and deepened my understanding through the Probability textbooks I used in my course at different times—books by Gnedenko, Meyer, Papoulis, Parzen, Pfeiffer, and Reza—as well as through interactions with my students. Discussions with colleagues at Syracuse University have also been helpful. Of these, I want to specifically acknowledge Prof. D. D. Weiner and Dr. M. Rangaswamy, who introduced me to spherically symmetric random variables; as well as Prof. F. Schlereth, who directed my attention to copulas.

However, the writing of this book could not have come to completion without the loving care and encouragement I received over all these years from my wife, Patricia Carey Schwarzlander. I also appreciate the understanding and support of my children, as well as my grandchildren, over the stressful period of nearly two years during which I converted the manuscript into publishable form and worked my way through the page proofs. I am also grateful for the assistance and encouragement provided by the staff of Wiley, Chichester. And I am indebted to the Department of Electrical Engineering and Computer Science at Syracuse University for having continued to provide me with an office, without which this project would have been very difficult to carry out.

Harry Schwarzlander

Syracuse, NY

November 2010

Introduction

Motivation and General Approach

The intent of this book is to give readers with an engineering perspective a good grounding in the basic machinery of Probability Theory and its application. The level of presentation and the organization of the material make the book suitable as the text for a one-semester course at the beginning graduate level, for students who are likely to have had an introductory undergraduate course in Probability. Nevertheless, the basic approach to applying probability mathematics to practical problems is not short-changed. Many years of teaching graduate students in Electrical Engineering have made it clear to me that it is very helpful to begin with a thorough treatment of the basic model, rather than gloss over this important material in favor of rushing on to a greater variety of advanced topics. This is the strategy pursued here.

Although the material is developed from first principles, a greater mathematical maturity is called for than is usually expected from most undergraduates, and intuitive motivational arguments are minimized. Nevertheless, portions of the book can be used for an undergraduate course, although it would differ significantly from a typical undergraduate ‘Probability and Statistics’ course. Furthermore, the material is presented in sufficient detail and organized in such a way that the book is also a very suitable text for self-study.

The book is unusual in a number of respects. It is intended for the student who applies probability theory. Yet there is not much discussion of specific applications, which would require much space to be devoted to establishing the specific problem contexts. Instead, it has been my intent to provide a thorough understanding (1) of probability theory as a mathematical tool, emphasizing its structure, and (2) of the underlying conceptual model that is needed for the correct application of that tool to practical problems. The manner in which a connection is made between a real-world problem and its mathematical representation is stressed from the beginning. On the other hand, not much emphasis is placed on combinatorics and on the properties of special distributions.

The degree of sophistication in probability expected of engineering students at the graduate level has changed over the years. In preparation for further study and research in stochastic processes, information theory, automata theory, detection theory, radar, and many other areas, familiarity with some of the more abstract topics, such as transformations of random variables, singular distributions, and sequences of random variables, is desirable. Such material is introduced with care at appropriate points in the book.

Throughout, my aim has been to help the reader achieve a solid understanding and feel comfortable and confident in applying probability theory in engineering research as well as in practical problems. Since most graduate students in engineering do not have the opportunity to go through two or three theoretical mathematics courses before starting into probability, this has meant placing more emphasis on introducing or reviewing various mathematical ideas needed in the development of the theory.

Organization of Material

The basic unit of presentation is the Section. Sections are numbered consecutively but are grouped into six major subdivisions—Parts I through VI—which could also be thought of as chapters. Each Section introduces and develops one or several related new concepts. There are problems associated with each Section and these are presented in the Appendix. Sections are somewhat self-contained, and the numbering of equations, definitions, theorems and problems begins a new in each Section, with the Section number as prefix.

To help the student reflect on the various concepts and become more familiar with them, each Section ends with one or more ‘queries,’ which are short review exercises. These also facilitate self-study. An answer table to the queries appears at the beginning of the Appendix.

Part I covers the development of the basic conceptual and mathematical models: the probabilistic experiment and the probability space. Devoting all of Part I to this introductory development was motivated by (1) my experience indicating that even students who do have an undergraduate probability background tend nevertheless to have a weak grasp of this material, and (2) the fact that it is an essential foundation on which to build a good understanding of more advanced concepts. For further emphasis, the first few Sections have intentionally been kept short.

In Part II there is brought together a mixture of topics pertaining to ‘elementary probability problems’— that is, to problems formulated in terms of a basic probability space without requiring the introduction of random variables. This portion of the book provides a bridge between Part I and the introduction of random variables in Part III. Some of the material is relatively simple, such as sampling with and without replacement. It is included for the sake of completeness, and some of this might be skipped in a graduate-level course. On the other hand conditional probability, independence, and product spaces are of course essential.

Part III serves to introduce the basics of random variables and an induced probability space. Discrete and absolutely continuous random variables are clearly distinguished. Discussion of multidimensional random variables has been divided into two steps. First, only two-dimensional random variables are introduced, whereas higher-dimensional random variables are discussed in Part IV. The reasons for this are that (1) it allows the student to first get a good grasp of the two-dimensional case, which is more easily visualized, and (2) it allows pursuit of a somewhat simplified syllabus from which higher-dimensional random variables are omitted. From Part III onward, some special attention is devoted to Gaussian distributions and Gaussian random variables because of their mathematical importance as well as their significance in engineering applications.

Transformations of random variables are addressed in considerable detail in Part IV. In addition to higher-dimensional random variables, Part IV also covers a number of other topics that do not require reference to expectations and moments, which are dealt with extensively in Part V. Delaying this important material to a later portion of the book is a choice I have made for pedagogic reasons. I believe it is helpful to gain a thorough grasp of the basic mechanics of random variables before getting involved in the various descriptors of random variables. Naturally, an instructor using this book can bring in some of that material earlier, if desired. Part V concludes with an introduction to the Riemann–Stieltjes integral, which allows some of the definitions in Part V to be extended to singular random variables. This has made it possible to bypass a discussion of the Lebesgue integral.

Part VI begins with a Section on complex random variables and then introduces the characteristic function and the generating function. There is a discussion of multidimensional Gaussian and Gaussian-like (‘spherically symmetric’) random variables. And following up on the Section on ‘entropy’ in Part II, this measure of randomness is now applied to random variables. Also introduced is the relatively new topic of ‘copulas.’ Then, sequences of random variables are treated in some detail, leading to the laws of large numbers and the Central Limit Theorem.

Notation and Terminology

Specialized notation and abbreviations have been avoided as much as possible. Frequent reference to a ‘probabilistic experiment’ has made the abbreviation ‘p.e.’ convenient. For intervals I have found Feller's [Fe2] overbar notation useful in order to avoid a proliferation of parentheses and brackets (see Section 36). It has been my experience that students readily accept this. A table summarizing all abbreviations and notation is provided in the Appendix.

Representative Syllabi

For a one-semester graduate course, the following is one way of selecting material from this book and grouping it into twelve weekly assignments:

For an undergraduate course, the book would be used in a different way, with greater emphasis on the development of the basic conceptual and mathematical ideas and on discrete probability. In this case, the following path into the book might be found useful:

A flow diagram depicting the prerequisite relationships among Sections is shown in Figure F.1. A reader who is interested primarily in studying a particular Section can see from this diagram with what background material he or she should be acquainted.

Figure F.1 Prerequisite relations among Sections

Part I

The Basic Model

Part I Introduction

This First Part is devoted to the development of the basic conceptual and mathematical formulation of a probability problem. The conceptual formulation serves to build up a clear and consistent way of thinking about a problem situation to which probability analysis is to be applied. The mathematical formulation is then constructed, step by step, by connecting set-theoretic concepts to the conceptual building blocks, and the relevant notation is introduced. Also, those principles of Set Theory that are needed for the mathematical formulation are presented in detail. The result is the ‘basic model’, which consists of a conceptual part (the ‘probabilistic experiment’) and a mathematical part (the ‘probability space’). Part I ends with some preliminary exercising of this model, and the establishment of several simple rules of probability arithmetic.

For the sake of emphasis, the individual Sections of Part I have intentionally been kept short. This is also intended to help ease the reader into a somewhat faster pace in the remaining Parts. Readers who have some familiarity with Probability principles should not skip Part I, but may find that they can assimilate the material presented here more quickly than those without that background.

About Queries:

Queries appearing at the end of a Section are short-answer review exercises that are intended to be worked out, pencil in hand, after studying that Section. A table of answers to queries appears in the Appendix. The number in brackets at the end of a query is the key for locating the answer in the answer table.

About Problems:

There are problems associated with each Section. These are located in the Appendix, arranged by Section.

Section 1

Dealing with ‘Real-World’ Problems

Probability theory is a branch of mathematics that has been developed for more effective mathematical treatment of those situations in the real world that involve uncertainty, or randomness, in some sense. Most nontrivial real-world problems incorporate just such situations, so that the practical importance of probability theory hardly needs to be stressed. It turns out, however, that intensive study of the theory alone can still leave the would-be practitioner peculiarly inept in its application. Why should this be so?

It is not inappropriate to ask this question right at the outset. After all, when commencing the exploration of new territory, it helps to be alerted about the nature of the obstacles that lie ahead. A few reflections can lead to an answer to our question.

Example 1.1

A young man working as a sales clerk in a furniture store is asked to count all the chairs that are on display in the store. These chairs may differ from each other in design and material. Nevertheless, the sales clerk automatically adds to his count each time his gaze comes to rest on an object that, in fact, anyone else would also regard as an instance of ‘chair’. This is because the concept ‘chair’ is firmly established in his mind. Possession of the concept ‘chair’ permits the clerk in this situation to carry out the simple mathematical operation of counting. Furthermore, the widely shared agreement regarding the concept ‘chair’ assures that the count will be the same as that obtained by anyone else, barring oversights or counting errors. We may say that with the aid of the concept ‘chair’, the clerk avoids conceptual errors in his task, although not necessarily mathematical errors, or procedural errors.

We can imagine how the clerk in this Example has his store environment modeled in terms of a large variety of concepts, some as simple and common as ‘chair’. These concepts serve to classify the environment, and act as discriminators in simple tasks such as discussed in the Example. On the other hand, it is important to realize that the applicability of a concept is not always clear-cut; concepts have a certain ‘fuzziness’. Surely we can devise an object that neither obviously is a chair, nor obviously isn't a chair.

In Example, the sales clerk is performing a ‘real-world’ task. He is concerned with the immediate experience of his store environment, to which he is applying his counting ability. This is the sense in which we will use ‘real world’. A person who picks up two dice and throws them acts in the real world, in contrast to another person who merely talks about throwing two dice.

Whenever mathematics is applied to a real-world problem, an important step is the establishment of a suitable model, or idealization, of the physical problem under consideration. This makes possible the application of the logical rules of mathematics to the real world. When we have a very simple problem, the modeling seems to take place rather automatically, so that we may not even be aware of it, as is surely the case with the sales clerk in Example 1.1. In more sophisticated situations, more effort generally goes into the modeling process. Because this modeling process always involves ‘conceptualization’—fitting the real-world problem at hand into our way of thinking about things—we will call such models conceptual models.

Example 1.2

An electronic engineer is handed a small cylindrical capsule with a short wire protruding from each end. He tries to ‘identify’ this object as a particular kind of electronic component, such as a resistor, or a capacitor. Actually, ‘resistor’, ‘capacitor’, etc., are conceptual models for real-world objects—the engineer's way of viewing the great variety of electronic components he encounters. His task is, therefore, to determine which of these models is most appropriate for the object at hand.

The model called ‘resistor’, for instance, pertains to real-world objects having two (or more) distinct places for electrical connections. Furthermore, the applicability of this model depends on the effect produced by the component in question when incorporated into an electric circuit. This effect depends on the internal construction of the component.

We noted before that a conceptual model makes possible the application of mathematics to the real world. In the situation just described it is the mathematics of electrical network theory. For instance, the conceptual model ‘resistor’ implies to the engineer that under suitable conditions the relationship

(1.1) equation

adequately characterizes the current i through the device in question, at an instant when the voltage across the terminals is e, where k is a proportionality constant. The conceptual model ‘resistor’ can therefore be considered as a bridge between a real-world object and the mathematical model that is expressed by the relation (1.1).

Of course, for a given real-world problem, the choice of a suitable model need not be unique, and the mathematical result obtained may depend on the particular model used. The appropriateness of any one model can be judged by how well the results obtained with it agree with, or predict, the actual physical nature or behavior of the situation being considered.

‘Networks’ are the conceptual models to which Network Theory is applied. Network Theory is not used in building the model, when some real-world electrical circuit is analyzed. But, once a model has been decided on, then Network Theory allows various conclusions to be drawn because it provides rules for applying abstract mathematical techniques to the model. The same process can be discerned whenever mathematics is applied to real-world problems. The application of Probability Theory to situations of randomness is also made through suitable conceptual models. However, as we will see in Section 2, the conceptual models required for probability problems are considerably more complicated than those needed in connection with most other branches of applied mathematics.

Just as Network Theory cannot be applied where there are no network models, so is Probability Theory useless without appropriate models. The establishment of an adequate model is therefore an essential ingredient in tackling any real-world problem by means of Probability Theory. Neglect of this important point easily results in confused and erroneous applications of the theory. Here lies the answer to the question posed at the beginning of this Section¹.

As we pursue our study of Probability in this book, we will try to pay attention to the connection between real-world problems and their mathematical formulation from the start. This will be possible with the help of a standard scheme for building conceptual models—a scheme that can be applied to all probability problems.

1. For further reading on the role of models in Applied Mathematics and the Sciences, see for example [Fr1], [Ri1], [MM1].

Section 2

The Probabilistic Experiment

To begin with, a name should be given to situations of uncertainty in the real world to which we will apply Probability Theory. The word ‘experiment’ suggests itself when thinking of various real-world situations that involve uncertainty:

the throwing of dice;

the measuring of a physical parameter, such as length, temperature,

or magnetic field strength;

sampling a batch of manufactured items.

We want to be able to express quantitatively the likelihood with which particular results will arise in situations such as these. But before we can even begin to apply any mathematics to such problems, we must have a clear picture of what it is we are actually dealing with. Now, ‘experiment’ is a very widely used word, and therefore imprecise in meaning. We will qualify it and will refer to any real-world ‘experiment’ as a ‘probabilistic experiment’ if it has been modeled in a manner that allows probability mathematics to be applied. It is also worth noting that, when trying to envision real-world situations such as referred to in the above list, we find that in each case there appears an observer—the experimenter—as an explicit or implicit part of the setting. This should not be surprising, since ‘uncertainty’ can only exist as the result of some sort of contemplation, and thus is really a state of mind—the state of mind of the experimenter.

Thus, the first question to be addressed is: what are the essential features of a suitable conceptual model—a probabilistic experiment? These are stated in the form of four distinct requirements in the Definition below, and are then illustrated by means of an Example. These requirements will assure that the model brings into focus those aspects of real-world ‘experiments’ that are essential to the correct application of the mathematical theory. In other words, we shall ‘talk probability’ (ask probability questions, and compute probabilities) only in those real-world contexts that we are able to model in the manner specified below. When this is not possible we do not have a legitimate probability problem. (We note that definitions within the conceptual realm lack the precision of a mathematical definition. As a reminder of this, the word ‘Definition’ appears in parentheses.)

(Definition) 2.1: Probabilistic experiment.

A probabilistic experiment is a conceptual model that consists of four distinct parts, as follows:

1.Statement of a purpose. By this is meant an expression of the intent to make specific real-world observations. It must specify a particular real-world context, configuration, or environment, which is pertinent to these observations. It must also include a listing or precise delineation of the various distinct ‘possibilities’ (properties, alternatives, facts, values, or observable conditions) that are considered to be of interest and that are to be watched for when the intended real-world observations are actually carried out.

2.Description of an experimental procedure. By experimental procedure is meant the specification of an unambiguous sequence of actions to be carried out by the experimenter, leaving no choice to the experimenter, and leading to the observations that are called for in the purpose.

3.Execution of the experimental procedure. This may only take place after the purpose has been stated and the description of the experimental procedure is completed.

4.Noting of the results. By this is meant the identification of those ‘possibilities’, listed as part of the purpose, which are actually observed to exist upon complete execution of the experimental procedure.

The following Example provides a first step toward understanding the significance of this Definition. Here, we model the throwing of a die as a probabilistic experiment.

Example 2.1: A die-throwing experiment.

1.The purpose of the experiment is to determine one of the following six possibilities that are observable upon throwing a given die, whose faces are marked in the standard way, onto a firm and reasonably smooth horizontal surface of adequate size to allow the die to roll freely and come to rest:

one dot faces up

two dots face up

......

......

six dots face up

2. The procedure is: The experimenter is to throw the die in the customary manner onto a surface such as specified in the purpose, with enough force so as to cause the die to roll. When the die comes to rest, the experimenter is to observe the number of dots facing up.

3.This procedure is carried out: An appropriate surface and a die are available. The experimenter throws the die in the specified manner. It comes to rest at a certain spot on the surface with various numbers of dots facing in various directions and, in particular, five dots facing upward.

4. Of all that is perceived by the experimenter upon carrying out the procedure—the path of the die, the final location or orientation of the die, the lighting conditions, the sound accompanying the throw of the die, etc.—only the property ‘five dots face up’ is one of the possibilities that was initially stated to be of interest. This property is noted.

This example is, of course, rather trivial. Nevertheless, it serves to illustrate the importance of each of the four steps making up the probabilistic experiment; for, knowledge of only one (or even two or three) of the four parts making up the model does not guarantee a unique description of the complete experiment. Thus, suppose that item 1 in the description of the die-throwing experiment were missing. From a knowledge of items 2, 3, and 4 it is not certain that 1 should read as it actually appears above. For instance, the purpose might actually have been to observe one of the properties

five dots face up

five dots face sideways

neither of the above.

It turns out that if this were the purpose, the probability problem arising from the experiment would be quite different.

Now suppose the description of the experimental procedure were missing. Even if item 3 is known to us—that is, we have seen the experimental procedure being carried out—we cannot usually be sure what the procedure really was. Just consider the possibility of the following sentence added to item 2 in the above Example: ‘If on the first throw the number of dots facing up is six, the die is thrown again.’ There would be no way of inferring this from a knowledge of items 3 and 4 alone.

Without knowledge of item 3, on the other hand, it is not clear whether properties noted under item 4 are observed as a result of carrying out the procedure as specified in item 2, or some other sequence of actions. It might also be proposed that items 2 and 3 be combined; however, there is a definite reason for maintaining them as separate items: The description of the experimental procedure must not change during the course of the experiment—that is, while the procedure is being carried out. Another way of stating this is to say that item 2 is strictly ‘deterministic’, whereas item 3 brings into evidence occurrences and properties about which there is initial uncertainty.

Finally, it is clear that an essential part of the experiment is missing if knowledge about the observed properties of interest, as specified in item 4, is not available. This situation will be considered again in Section 3.

Thus, we have seen that even as innocent an activity as throwing a die calls for a careful description if it is to be modeled as a probabilistic experiment. Even more care will be needed when dealing with more complicated experiments, but this can only be appreciated after our study has progressed.

Particular attention has to be given to the purpose of a probabilistic experiment. The purpose cannot be in the form of a question, for instance. Consider: ‘Will hydrochloric acid produce a precipitate in the unknown liquid sample?’ This sentence does not conform with item 1 of our Definition and therefore cannot be the statement of purpose of a probabilistic experiment; it gives no clear account of the possible alternatives that are to be looked for in this experiment. Also, a probabilistic experiment cannot have as its purpose the determination of a probability. At this point we have not yet assigned a technical meaning to the word ‘probability’. We are not yet concerned with probability, only with experiments. But if we stick to the common language sense of ‘probability’, which we might paraphrase as ‘likelihood’, or ‘chance’, this surely is not a property that is observable in the real world as a result of performing an experiment. Similarly, the purpose of a probabilistic experiment cannot be to decide something. Decisions might be made on the basis of the results of the experiment, but this circumstance does not enter into our model.

Probability Theory is actually not concerned with completed probabilistic experiments—only with probabilistic experiments prior to their execution. Nevertheless, a picture of the complete probabilistic experiment must be in our mind, as will become clearer as we proceed. We will sometimes refer to a probabilistic experiment prior to execution—i.e., to parts 1 and 2 of the Definition—as the ‘experimental plan’.

The discussion so far leads us to view our involvement in a real-world problem as taking place in three different domains, as illustrated in Figure 2.1. In the external world or ‘Real World’, a person experiences a profusion of perceptions. The ‘Conceptual World’ provides the opportunity to clarify, organize and prioritize these perceptions. Furthermore, it forms the bridge between real-world experience and mathematical analysis. In the Conceptual World, as mentioned earlier, ‘definitions’ are of a different kind from those in the Mathematical World. They generally are rules for relating the real world to the conceptual world, and thus are susceptible to interpretation. There is usually a fuzziness to these definitions, that is, their application to any given problem is not always entirely clear-cut. It is then necessary to use judgment based on experience, or to proceed with caution and be prepared to modify one's approach.

Figure 2.1 Applying mathematics to real-world problems

The notion of probability evolved in the conceptual world. It does not exist in the real world. Designating our conceptual experimental model a ‘probabilistic experiment’ therefore seems fitting. The definition of a probabilistic experiment is our first encounter with a ‘definition’ in the Conceptual World. We may think of it also as a rule for constructing the right kind of conceptual model for any given real-world experiment—a rule telling us how to organize our thinking about the experiment. A probabilistic experiment will always serve as our conceptual model when we try to apply Probability Theory to real-world problems, as indicated in Figure 2.1. Probability Theory may not lead to meaningful results when dealing with a real-world problem that cannot be modeled in accordance with Definition 1 or some other suitable conceptual model.¹

Real-world problems that can be modeled as probabilistic experiments have a peculiar feature that is not shared by most other types of problems to which Mathematics can be applied. It is the fact that a human being, an observer—the experimenter—is an essential part of the model. The experimenter's mind harbors the uncertainty (or certainty) about the properties that will be observed in an experiment. Thus, Probability Theory is unique, since it permits the application of mathematical techniques to a class of problems that in a certain sense include the interaction of a human being with his/her environment.

Queries

Note: Queries appearing at the end of a section are to be considered part of the material of that section. The reader should answer them, pencil and paper at hand, in order to assure a correct understanding of the ideas presented. The number in brackets following each Query identifies the answer in the Answer Table (see Appendix).

2.1 Which of the following are correct statements, which incorrect, based on a reasonable interpretation of the definition of a probabilistic experiment? If incorrect, which part(s) of Definition 2.1 are violated?

a. A person playing a game of checkers is conducting a probabilistic experiment.

b. A person, prior to playing a game of checkers, is deciding on a strategy. This constitutes a probabilistic experiment.

c. A person decides to take a walk along a certain stretch of seashore to see if she might find something interesting, such as driftwood, shells, etc. She is embarking on a probabilistic experiment.

d. A person is waiting at Kennedy airport for a friend who is to arrive on a particular flight. She wonders whether or not the friend will arrive safely. She is engaged in a probabilistic experiment.

e. A person with high blood pressure is about to fly from New York to London. He wonders whether he will get there or whether he will die from a heart attack or perish in a crash along the way. He is engaging in a probabilistic experiment.

f. A mail clerk puts a letter on a letter scale to see whether it weighs more than an ounce or not. He is performing a probabilistic experiment. [184]

1. The Definition of this Section, and the additional requirements put on the conceptual model in Section 15, follow closely the specifications presented in [AR1]. Other ‘systems’ of probability exist, which are not built on the particular kind of conceptual model introduced here. This fact can be a source of confusion, because nearly the same terminology and mathematics appears in all of them. Our approach here is along those lines that are now most widely accepted, especially in the physical and natural sciences and in engineering.

Section 3

Outcome

The ‘probabilistic experiment’ (henceforth abbreviated p.e.) is the conceptual model through which we will view any real world problem to which we want to apply Probability Theory. As we proceed, we will become quite accustomed to working with p.e.'s. We will also explore some of the difficulties that can arise when the conceptual model is bypassed. But before establishing connections to mathematics, a little more needs to be said about the model.

(Definition) 3.1

Of the distinct possibilities of interest in a particular p.e., all those that are actually observed after the procedure has been carried out are collectively called the actual outcome (or simply, outcome) of the p.e.

For instance, the die-throwing experiment in Example has as the actual outcome the single property ‘five dots face up‘. But an outcome can also consist of several properties, all of which are observed upon executing the procedure. On the other hand it follows from the above Definition that an experiment cannot have more than one outcome. An experimental plan leads to one actual outcome from among a variety of candidates, upon completion of the experiment. These various candidates are referred to as the possible outcomes of the p.e., or of the experimental plan. Thus, the die-throwing experiment of Section 2 has six possible outcomes, and each happens to be associated with exactly one of the six properties that are of interest.

It is often convenient to incorporate the notion of ‘possible outcomes' in the description of a p.e.: In the statement of purpose of a p.e., we can replace the list of ‘possibilities' or ‘properties', etc., by a listing of all the possible outcomes. In our die-throwing experiment this happens to cause no change, since the possible outcomes coincide with the various properties of interest. This is not always the case.

Example 3.1: Throw of two dice.

An experiment similar to the one modeled in Example is to be performed with two dice—one red and one white die. There are now 12 properties of interest:

one dot faces up on red die

. . .

. . .

six dots face up on red die

one dot faces up on white die

. . .

. . .

six dots face up on white die

On the other hand, this p.e. has 6 × 6 = 36 possible outcomes, covering all possible combinations of results on the red and the white die. Thus, each possible outcome is made up of two ‘properties of interest'. In the statement of purpose for this p.e., it is simpler to list all properties of interest than all possible outcomes.

Now let us suppose the die-throwing experiment is performed on a sidewalk and the die falls down a drain. How does this situation fit into the schema of Definition 2.1? None of the properties of interest in the experiment can be observed in this case, so that there is no outcome! Although this experiment may have been ‘properly' performed, in the sense that the experimenter carried out all the required actions, it nevertheless turned out in an undesirable manner. We say that such an experiment, which has no outcome, was unsatisfactorily performed.

Henceforth, unsatisfactorily performed experiments will be excluded from our considerations. The theory to be developed, which will allow us to treat mathematically those situations in which observables arise with some degree of uncertainty, presupposes experiments which are modeled according to Definition 2.1—i.e., as probabilistic experiments—and furthermore, these experiments must be satisfactorily performed. This should not be regarded as restricting the applicability of the theory; it merely refines our conceptual model. For, we have in fact two ways of accommodating the situation described above, where a die falls down a drain:

a. As an unsatisfactorily performed experiment it does not exist in our conceptual model world—we simply do not think of the actions that lead to the loss of the die as belonging to a p.e. Or:

b. As an experiment whose possible outcomes include ‘die is lost' or some such property, it does have an acceptable representation in our conceptual model world. In this way it can be arranged that an otherwise unsatisfactorily performed experiment gets treated as a satisfactorily performed one.

In practice, any situation involving uncertainties can be framed within some kind of hypothetical experiment to which the scheme of Definition 2.1 can be applied. Often, several different possible experiments suggest themselves. Each of these different formulations may result in a different mathematical treatment of the problem at hand.

Queries

3.1 Consider various p.e.'s whose procedure calls for one throw of an ordinary die onto a table top. For each of the following lists, decide whether it can be a listing of the possible outcomes of such an experiment.

a.‘one dot faces up'

‘two dots face up'

‘three dots face up'

‘four dots face up'

‘five dots face up'

b. ‘three or fewer dots face up'

‘an even number of dots faces up'

‘four or more dots face up'

c. ‘one dot faces up'

d. ‘an even number of dots faces up'

‘one dot faces up'

e.‘die comes to rest on table top'

‘die falls off table'

f.‘the experiment is satisfactorily performed'

‘the experiment is unsatisfactorily performed'. [42]

3.2 In the statement of purpose of a certain p.e. there appears a list of exactly three observable properties, or features, which are to be looked for upon executing the procedure. Allowing only satisfactorily performed experiments, what is the maximum number of possible outcomes that might exist for this experiment? The minimum number? [226]

3.3 The purpose of a particular p.e. includes a listing of various observable features, or properties, that are of interest in this experiment and that are to be looked for upon executing the procedure. What is the least number of observable features that must be in this listing if it is known that the p.e. has five possible outcomes? [195]

3.4 An experimenter who is about to perform a particular p.e. claims that she knows which one of the possible outcomes will be the actual outcome, prior to performing the experiment. This implies which of these:

a. She will see to it that the experiment will be unsatisfactorily performed

b. The experiment is ‘rigged'

c. The experiment has only one possible outcome

d. She has performed the experiment previously. [213]

Section 4

Events

Often, we will be especially interested in some group or collection of possible outcomes from among all the various possible outcomes of a p.e. We may then wish to express the fact that the actual outcome, when the experiment is performed, belongs to this particular collection of possible outcomes. Consider again the die-throwing experiment with possible outcomes ‘one dot faces up’, ‘two dots face up’, … ‘six dots face up’. We might be particularly interested to see whether the actual outcome of this experiment is characterized by the description

(4.1) equation

This description encompasses several possible outcomes, namely:

‘two dots face up’

‘four dots face up’

‘six dots face up’.

(Definition) 4.1

Suppose that an experimental plan and a listing of the possible outcomes have been specified. Any characterization of results that might be observed upon performing the experiment is called an event. An event therefore represents some collection of possible outcomes. If the experiment is performed and its actual outcome belongs to an event that has been specified for this experiment, then that event is said to have occurred.

Statement 4.1, above, is an example of an event that can be defined for the die-throwing experiment. We see that the description of an event need not explicitly identify all the possible outcomes that it encompasses.

For a given p.e. or experimental plan we may find it convenient to define several events, in order to simplify various statements about the experiment. Using again the die-throwing experiment as an example, we may be interested in stating whether, upon performing the experiment, more than three dots face up. Thus, we define for this experiment another event

‘more than three dots face up’

which consists of the possible outcomes

‘four dots face up’

‘five dots face up’

‘six dots face up’.

In the die-throwing experiment described in Example, the actual outcome was ‘five dots face up’, so that the event ‘more than three dots face up’ did occur, but not the event ‘an even number of dots is showing’.

There is, of course, another way of expressing whether a throw of the die results in more than three dots facing up. That is to revise the model by defining the probabilistic experiment in such a way that ‘more than three dots face up’ is one of the possible outcomes. Then we would be able to say that, in Example 2.1, the outcome was ‘more than three dots face up’. In typical applications, however, this approach is cumbersome: once a fairly straightforward model has been established, it is usually preferable to stick with it.

It should be realized that the notion of ‘event’, as we have defined it, exists only in our conceptual model world. It has no counterpart in the real world in the absence of a conceptual model. It arises from the way in which we have decided to think about situations of uncertainty in the real world. Naturally, our specific use of the word ‘event’ must not be confused with its use in ordinary language, where it is synonymous with ‘happening’, or ‘occurrence’.

Before concluding this Section we note some relationships between experiments, possible outcomes, and events.

a. Whenever we speak of events and possible outcomes, a particular experimental plan is assumed to underlie the discussion, even if it is not explicitly stated.

b. If two events are defined in such a way that both represent the same collection of possible outcomes, then they are considered to be the same event, even if their verbal descriptions differ.

c. An event can be defined in such a way that it comprises only one of the possible outcomes. Such an event is called an elementary event.

d. An event can be defined in such a way that it comprises none of the possible outcomes (the particular collection of outcomes that is empty). Such an event cannot occur when the experiment in question is performed (since we do not allow an experiment to have no outcome), and is therefore called an impossible event.

e. The event that comprises all possible outcomes of a particular p.e. is called the certain event, since it must occur when the experiment is (satisfactorily) performed.

f. Several different events may occur when an experiment is performed, even though an experiment always has only one outcome.

g. There may be uncertainty as to which of several possible outcomes will be the actual outcome when an experiment is about to be executed. But after the experiment has been performed, all uncertainty has been removed. It must then be perfectly clear what the actual outcome is; and whether or not a particular event has occurred.

Queries

4.1 Consider two different die-throwing experiments:

p.e. no. 1 has these possible outcomes:

‘one or two dots face up’

‘three or four dots face up’

‘five or six dots face up’.

p.e. no. 2 has these possible outcomes:

‘an odd number of dots faces up’

‘an even number of dots faces up’.

For each of the following statements, indicate whether it characterizes an event in no. 1, an event in no. 2, in both, or in neither:

a. more than two dots face up

b. an odd number of dots faces up

c. one dot faces up

d. one or more dots face up

e. no dots face up. [117]

4.2 For a particular p.e., three events are specified, no two of which are the same.

a. What is the minimum number of possible outcomes this p.e. must have?

b. Same question, if it is known that all three events can occur together? [19]

Section 5

The Connection to the Mathematical World

Our next concern is the connecting link between the conceptual model and the mathematics of probability. We developed our conceptual model to the point where we talked about collections of outcomes—namely, events. It is therefore natural to turn to the branch of mathematics called Set Theory, which is a completely abstract theory applicable to problems that involve collections of objects. Two basic features of Set Theory are:

a. It requires well-defined collections of objects: A collection of objects is well defined if, given any object, this object either is a member of the collection, or it is not a member of the collection—it cannot be both.

b. There exists no uncertainty, vagueness or ambiguity. It is a strictly deterministic theory.

We may best grasp the significance of these two features by trying to think of examples of collections that lack them.

Example 5.1

Here is a collection that is not well-defined: ‘Of all the children registered this year in a particular school, the collection of all those who are nine years old.’ This collection is not well-defined because at different times of the year, not all the same children will be nine years old. Besides, some children may transfer into or away from the school during the year.

Example 5.2

We often make reference to collections of objects that are governed by some degree of uncertainty. Consider a ‘bridge hand’. In the game of bridge, 13 cards are dealt to each of the four players, from a shuffled deck of 52 different cards. When we speak of a ‘bridge hand’, we are referring to a collection of 13 cards such as might be dealt to one of the players, without specifying which particular cards they are.

Looking now at the conceptual model we have developed so far, it should be clear that we have carefully constructed it in such a way that properties (a) and (b), listed above, are satisfied if the ‘objects’ in question are taken to be the possible outcomes of a probabilistic experiment.

We note the following set-theoretic terms:

(Definition) 5.1

a. A set is a well-defined collection of abstract objects.

b.Elements are the basic objects we are dealing with.

c. A space is a nonempty set; that is, a set containing at least one element. Usually we mean by space that particular set that contains all the elements that are defined in a given problem or discussion.

The elements of a set, as abstract mathematical objects, are considered devoid of attributes, but are distinct. This seems contradictory since the absence of attributes would seem to make elements indistinguishable. We get around this by thinking of each of the elements of a set as representing a conceptual object so that, in particular, we can label and identify the individual elements of a set.

A set of elements, therefore, is the mathematical model for a collection of conceptual objects (or for the corresponding real-world objects), which abstracts the distinctness of the objects and their belonging to the collection, nothing else. A set of elements may also serve as the model for a collection of other mathematical objects (with mathematical attributes)—for instance, the collection of integers from 1 to 6. In such a case it becomes tedious and also unnecessary to maintain the distinction that a ‘set of six elements’ is a model for the collection of integers from 1 to 6. We speak then simply of ‘the set of integers from 1 to 6’, or the like.

It should now be clear that our earlier discussion of the conceptual model needs to connect with the mathematical entities set, element and space in the manner that is illustrated in Figure 5.1.

Figure 5.1 Making connections between the conceptual and mathematical worlds

The naturalness of this correspondence accounts for an easy diffusion of terminology across the boundary separating the conceptual and the mathematical world. Also, the possible outcomes are often called samples, and the elements points; these terms are often combined to give sample points, and it has become customary to refer to the collection of all possible outcomes of an experiment as the space of outcomes, or sample space. On the other hand, ‘sample space’ and ‘event’ are also used in Set Theory when this theory is applied to probability problems.

If the correspondence between the conceptual and mathematical worlds is made carelessly, inconsistencies can arise that will result in paradoxical results. Such difficulties have existed throughout much of the history of Probability. Establishing a firm correspondence between the conceptual and mathematical world is no trivial matter. Besides permitting the use of mathematics for the solution of real-world problems, it can lead to a better understanding of the conceptual model. It may also lead to the introduction of additional constraints into the conceptual model that are inherent in the mathematical structure. For example, we have spoken of impossible events. Conceptually it may seem appropriate to consider a variety of different impossible events, in connection with a given probabilistic experiment. In the next Section we see that, as a result of the particular way in which we make the connection to the mathematical world, these various impossible events become indistinguishable. Our mathematical model allows only a single impossible event.

In the next few Sections we will explore those elementary notions of Set Theory that pertain to the mathematics of Probability.

Queries

5.1 Which of the following collections can be modeled by a set of elements?

a. All the chairs on display in a particular furniture store.

b. Consider a die-throwing experiment that has not yet been performed. The collection of those attributes of interest in this experiment that will not be observed upon performing the experiment.

c. The collection of all mathematical theorems that have not yet been discovered (i.e., never stated, not even as conjectures).

d. The collection of all the different sets that can be formed from elements in a given set.

e. The collection of all aqueous solutions in which a drop of concentrated hydrochloric acid produces a precipitate at 20 °C.

f. The collection of all the meanings attributable to some word that is given. [36]

5.2 Consider the die-throwing experiment of Example 2.1. Let the collection of all possible outcomes be represented by a space of elements. For each of the following, state whether it is represented by an element, a set, or neither.

a.‘the experiment has not been performed’

b.‘the experiment has been performed’

c.‘more dots face up than the next time the experiment will be performed’

d.‘one dot faces up’

e.‘more than one dot faces up’

f.‘more than six dots face up’. [122]

Section 6

Elements and Sets

It is worth repeating that Set Theory is concerned with well-defined collections of objects. In any problem involving Set Theory, one well-defined collection of objects must always be the collection of all the elements (objects) that can arise in that problem. In other words, the problem statement must give enough information to define the set of all the elements that are under consideration in the particular problem. As mentioned in Section 5, we call this set the space, or sample space, of the problem; and we denote it S.¹ If the space is not defined, Set Theory can lead to inconsistent results.

Once the space S is defined for a given problem, then it must be true that any element ξ arising in that problem belongs to (or ‘is an element of’) the particular set that serves as the space S. This relation is expressed symbolically by

Henceforth, a space S will always be assumed to have at least two elements.

(Definition) 6.1

Given a space S and sets A, B. Then A is a subset of B, denoted

if and only if ξ ∈ A implies ξ ∈ B, for all elements ξ (i.e., for all elements ξ ∈ S).

For example, if a space S is defined for a given problem, then for any set S that arises in the problem the statement A ⊂ S must be true. The statements A ⊂ B, B ⊃ A are also read ‘A is contained (or included) in B’, ‘B contains (or includes) A’.

The relationships expressed by ∈ and ⊂ are easily visualized in pictorial form by means of a so-called ‘Venn diagram’, as shown in Figure 6.1. In this diagram the large rectangular outline encloses a region that symbolizes the collection of all the points making up the space S for a given problem. Any point in this region then represents an element in S, that is, an element that can be talked about. Collections of such elements—that is, sets—are then represented by suitable regions within the area allotted to S. Individual elements are sometimes indicated by dots. When the region representing a set A is drawn inside the region representing another set B, as in Figure 6.1, this is meant to indicate that A ⊂ B.

Figure 6.1 Venn diagram

The above notions relate readily to our conceptual model. The actual outcome of a p.e. must always be one of the possible outcomes. In set theoretic terms this means: Whenever we are modeling a p.e., every sample point that pertains to this p.e. must belong to the sample space for the p.e. Also, all the events that can be talked about consist of possible outcomes that belong to the sample space. Therefore, we may bring some Set Theory terminology into our conceptual model and say that these events are ‘contained in’ the collection of all possible outcomes, the sample space. For any given problem, we can think of the sample space as specifying a kind of ‘universe’. All manipulations we might perform on its contents must keep us within its domain of definition.

The difference in the meanings of the two symbols ∈, must be kept in mind. The symbol ∈ relates an object to a collection of such objects. The symbol relates two things of like category, namely, two collections of similar objects. This is the way the symbols ∈, are defined, so that their use in any other way is meaningless. The following statements, implied by Figure 6.1, illustrate the correct usage (where ∉ and ⊄ are negations of ∈ and , respectively):

We have used letter symbols A, B, C to identify sets. A set may also be identified by listing all its elements and enclosing this listing between braces. If ξa, ξb denote two distinct elements, then {ξa, ξb} is the set whose elements are ξa and ξb. Braces will always be read ‘the set whose elements are (contents of brace)’. Rather than enumerating all the elements that make up a set, we may characterize them in some way. Thus, the expression { } stands for ‘the set of all ξ's such that ξ is not an element of A’. Another way of saying it is ‘the set of those ξ's that satisfy the condition ‘ξ is not an element of A'.'

It is important to distinguish between an element ξ and the set whose only element is ξ, denoted {ξ}. Thus, ξ ∈ {ξ} and {ξ} ⊂ S; but the following statements are meaningless: ξ {ξ}, {ξ} ∈ S. This illustrates how the details of the mathematical model can force additional constraints onto the conceptual model. Because of the manner in which we established the correspondence between the conceptual model and Set Theory, we must make a careful distinction between a possible outcome and the event consisting of only that one possible outcome. The former corresponds to an element in the space S, the latter to a subset of the space S.

Finally, it is possible to speak of a collection that is entirely devoid of objects. Thus, we have an empty set, which is denoted ϕ. Since there are no elements ξ belonging to ϕ, then in Definition 6.1 the requirement stated there is satisfied ‘vacuously' if A stands for ϕ. Thus, ϕ ⊂ B is true for every set B ⊂ S. Note that corresponding to the empty set of Set Theory we have in our conceptual model an ‘impossible event'.

Above we defined the relation ‘ ' between sets, also called the ‘inclusion relation'. Another important relation between sets is equality.

(Definition) 6.2

Given a space S and sets A, B. Then A and B are the same set, or are equal, denoted

if and only if A ⊂ B and also A ⊃ B.

It follows that if A is an empty set and B is an empty set, then A = B; i e., there is only one empty set ϕ ⊂ S. Here again, the mathematical model influences the conceptual model. While we might envision many different kinds of impossible events in connection with a particular p.e., we will speak of only one impossible event (as noted in Section 5) since there is only one empty set in the mathematical model.

If two sets A, B are not the same set, i.e., are not equal, then they are called distinct and we write A ≠ B.

Queries

6.1 A space S is given, with subsets A, B. Also, let