To HARRIET To my GRANDCHILDREN To JOAN, LISA, and KARIN

A.M.M. F.A.G. D ..C.B.

Library of Congress Cataloging in Publication Data Mood, Alexander McFar1ane, 1913Introduction to the theory of statistics. (McGraw-Hi1l series in probability and statistics) Bibliography: p. 1. Mathematical statistics. I. Graybill. Frank1in A., joint author. II. .Boes, Duane C., joint author. III. Title. QA276.M67 1974 519.5 73-292 ISBN 0-{)7-042864-6

INTRODUCTION TO THE THEORY OF STATISTICS

Copyright 1963. t 974 by McGraw-Hill, Inc. All rights reserved. Copyright 1950 by McGraw-Hili, Inc. All rights reserved. Printed in the United States of America. No part of this pubJication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic. mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher.

6789 10 KPKP 7832109

This book was set in Times Roman. The editors were Brete C. Harrison and Madelaine Eichberg; the cover was designed by Nicholas Krenitsky; and the production supervisor was Ted Agrillo. The drawings were done by Oxford Il1ustrators Limited. The printer and binder was Kinsport Press, Inc.

CONTENTS

Preface to the Third Edition Excerpts from the First and Second Edition Prefaces
I

Xlll

...

xv
1
1 2 2 3 5 8 8 9 14 19 25 32

Probability

1 Introduction and Summary 2 Kinds of Probability 2.1 Introduction 2.2 Classical or a Priori Probability
2.3 A Posteriori or Frequency Probability 3 Probability-Axiomatic 3.1 Probability Models 3.2 An Aside-Set Theory 3.3 Definitions of Sample Space and Event 3.4 Definition of Probability 3.5 Finite Sample Spaces 3.6 Conditional Probability and Independence

vi

CONTENTS

II Random Variables, Distribution Functions, and

Expectation
1 Introduction and Summary 2 Random Variable and Cumulative Distribution Function 2.1 Introduction 2.2 Definitions 3 Density Functions 3.1 Discrete Random Variables 3.2 Continuous Random Variables 3.3 Other Random Variables 4 Expectations and Moments 4.1 Mean 4.2 Variance .4.3 Expected Value of a Function of a Random Variable 4.4 Ch~byshev Inequali ty 4.5 Jensen Inequality 4.6 Moments and Moment Generating Functions

51

51 52 52 53 57 57 60 62 64 64 67 69 71 72 72

ill Special Parametric Families of Univariate Distributions

1 Introduction and Summary

85

2 Discrete Distributions 2.1 Discrete Uniform Distribution 2.2 Bernoulli and Binomial Distributions 2.3 Hypergeometric Distribution 2.4 Poisson Distribution 2.5 Geometric and Negative Binomial Distributions 2.6 Other Discrete Distributions 3 Continuous Distributions ).( Uniform or Rectangular Distribution (3.2 ) Normal Distribution '-3.3 Exponential and Gamma Distributions 3.4 Beta Distribution 3.5 Other Continuous Distributions 4 Comments 4.1 Approximations 4.2 Poisson and Exponential Relationship 4.3 Contagious Distributions and Truncated Distributions
# .

85 86 86 87 91 93 99 103 105 105 107 111 115 116

119

119 121 122

CONTENTS

vii

IV Joint and Conditional Distributions, Stochastic Independence, More Expectation

1 Introduction and Summary 2 Joint Distribution Functions 2.1 Cumulative Distribution Function 2.2 Joint Density Functions for Discrete Random Variables 2.3 Joint Density Functions for Continuous Random Variables 3 Conditional Distributions and Stochastic Independence 3.1 Conditional Distribution Functions for Discrete Random Variables 3.2 Conditional Distribution Functions for Continuous Random Variables 3.3 More on Conditional Distribution Functions 3.4 Independence 4 Expectation 4.1 Definition 4.2 Covariance and Correlation Coefficient 4.3 Conditional Expectations 4.4 Joint Moment Generating Function and Moments 4.5 Independence and Expectation 4.6 Cauchy-Schwarz Inequality 5 Bivariate Normal Distribution 5.1 Density Function 5.2 Moment Generating Function and Moments 5.3 - Marginal and Conditional Densities

129

129 130 130 133 138 143 143 146 148 150 153 153 155 157 159 160 162 162 162 164 167
175

V Distributions of Functions of Random Variables

1 Introduction and Summary 2 Expectations of Functions of Random Variables 2.1 Expectation Two Ways 2.2 Sums of Random Variables 2.3 Product and Quotient 3 Cumulative-distribution-function Technique 3.1 Description of Technique 3.2 Distribution of Minimum and Maximum 3.3 Distribution of Sum and Difference of Two Random Variables 3.4 Distribution of Product and Quotient

175 176 176 178 180 181 181 182 185 187

viii

CONTENTS

4 Moment-generating-function Technique 4.1 Description of Technique 4.2 Distribution of Sums of Independent Random Variables 5 The Transformation Y = g(X} 5.1 Distribution of Y g(X) 5.2 Probability Integral Transform 6 Transformations 6.1 Discrete Random Variables 6.2 Continuous Random Variables

189 189 192 198 198 202 203 203 204

VI Sampling and Sampling Distributions

1 Introduction and Summary 2 Sampling 2.1 Inductive Inference 2.2 Populations and Samples 2.3 Distribution of Sample 2.4 Statistic and Sample Moments 3 Sample Mean ~Mean and Variance 3.2 Law of Large Numbers .3 Central-limit Theorem 3.4 Bernoulli and Poisson Distributions 3.5 Exponential Distribution 3.6 Uniform Distribution 3.7 Cauchy Distribution 4 Sampling from the Normal Distributions 4.1 Role of the Normal Distribution in Statistics 4.2 Sample Mean 4.3 The Chi-square Distribution 4.4 The F Distribution 4.5 Student's t Distribution 5 Order Statistics 5.1 Definition and Distributions 5.2 Distribution of Functions of Order Statistics 5.3 Asymptotic Distributions 5.4 Sample Cumulative Distribution Function

219

~ 254
256 264

219 220 220 222 224 226 230 231 231 233 236 237 238 238 239 239 240 241 246 249

CONTENTS

ix

VII Parametric Point Estimation

1 Introduction and Summary 2 Methods of Finding Estimators 2.1 Methods of Moments 2.2 Maximum Likelihood 2.3 Other Methods
3

271

Properties of Point Estimators 3.1 Closeness 3.2 Mean-squared Error 3.3 Consistency and BAN 3.4 Loss and Risk Functions Sufficiency 4.1 Sufficient Statistics 4.2 Factorization Criterion 4.3 Minimal Sufficient Statistics 4.4 Exponential Family Unbiased Estimation 5.1 Lower Bound for Variance 5.2 Sufficiency and Completeness Location or Scale Invariance 6.1 Location Invariance 6.2 Scale Invariance Bayes Estimators 7.1 Posterior Distribution 7.2 Loss-function Approach 7.3 Minimax Estimator Vector of Parameters Optimum Properties of Maximum-likelihood Estimation

271 273 274 276 286 288 288 291 294 297 299 300 307 311 312 315 315 321 331 332 336 339 340 343 350 351 358

vm Parametric Interval Estimation

1 Introduction and Summary 2 Confidence Intervals 2.1 An Introduction to Confidence Intervals 2.2 Definition of Confidence Interval 2.3 Pivotal Quantity

372

372 373 373 377 379

CONTENTS

3 Sampling from the Normal Distribution 3.1 Confidence Interval for the Mean 3.2 Confidence Interval for the Variance 3.3 Simultaneous Confidence Region for the Mean and Variance 3.4 Confidence Interval for Difference in Means 4 Methods of Finding Confidence Intervals 4.1 Pivotal-quantity Method 4.2 Statistical Method 5 Large-sample Confidence Intervals
6

Bayesian Interval Estimates

381 381 382 384 386 387 387. 389 393 396

IX Tests of Hypotheses

401

1 Introduction and Summary 2 Simple Hypothesis versus Simple Alternative 2.1 Introduction 2.2 Most Powerful Test 2.3 Loss Function 3 Composite Hypotheses 3.1 Generalized Likelihood-ratio Test 3.2 Uniformly Most Powerful Tests 3.3 Unbiased Tests 3.4 Methods of Finding Tests 4 Tests of Hypotheses-Sampling from the Normal Distribution 4.1 Tests on the Mean 4.2 Tests on the Variance 4.3 Tests on Several Means 4.4 Tests on Several Variances
5 Chi-square Tests. 5.1 Asymptotic Distribution of Generalized Likelihood-ratio 5.2 Chi-square Goodnessof-fit Test 5.3 Test of the Equality of Two Multinomial Distributions and Generalizations 5.4 Tests of Independence in Contingency Tables 6 Tests of Hypotheses and Confidence Intervals 7 Sequenti'al Tests of Hypotheses 7.1 Introduction

401 409 409 410 414 418 419 421 425 425 428 428 431 432 438 440 440 442 448 452 461 464 464

CONTENTS

xi

7.2 Definition of Sequential Probability Ratio Test 7.3 Approximate Sequential Probability Ratio Test 7.4 Approximate Expected Sample Size of Sequential Probability Ratio Test

466 468 470

Linear Models

482
482 483 484 487 491 494 498

1 Introduction and Summary 2 Examples of the Linear Model

3 4 5 6
7

Definition of Linear Model Point Estimation~Case A Confidence Intervals-Case A Tests of Hypotheses-Case A Point Estimation-Case B

XI N onparametric Methods

504

1 Introduction and Summary 2 Inferences Concerning a Cumulative Distribution Function 2.1 Sample or Empirical Cumulative Distribution Function 2.2 Kolmogorov-Smirnov Goodness-of-fit Test 2.3 Confidence Bands for Cumulative Distribution Function 3 Inferences Concerning Quantiles 3.1 Point and Interval Estimates of a Quantile 3.2 Tests of Hypotheses Concerning Quantiles 4 Tolerance Limits 5 Equality of Two Distributions 5.1 Introduction 5.2 Two-sample Sign Test 5.3 Run Test 5.4 Median Test 5.5 Rank-sum Test

504 506 506 508 511 512 512 514 515 518 518 519 519 521 522
527

Appendix A. Mathematical Addendum

1 Introduction

527

xii

CONTENTS

2 Noncalculus 2.1 Summation and Product Notation 2.2 Factorial and Combinatorial Symbols and Conventions 2.3 Stirling's Formula 2.4 The Binomial and Multinomial Theorems 3 Calculus 3.1 Preliminaries 3.2 Taylor Series 3.3 The Gamma and Beta Functions
Appendix B. Tabular Summary of Parametric Families of Distributions
1 Introduction

527 527 528 530 530 531 531 533 534

537 537 538 540 544 544 544 545 545 545 546 546 546 547 548 548 548 548 549 549 550 557

Table 1. Discrete Distributions Table 2. Continuous Distributions

Appendix C. References and Related Reading

Mathematics Books Probability Books Probability and Statistics Books Advanced (more advanced than MGB) Intermediate (about the same level as MGB) Elementary (less advanced than MGB, but calculus prerequisite) Special Books Papers Books of Tables

Appendix D. Tables
1

Description of Tables Table 1. Ordinates of the Normal Density Function Table 2. Cumulative Normal Distribution Table 3. Cumulative Chi-square Distribution Table 4. Cumulative F Distribution Table 5. Cumulative Student's t Distribution

Index

PREFACE TO THE THIRD EDITION

The purpose of the third edition of this book is to give a sound and self-contained (in the sense that the necessary probability theory is included) introduction to classical or mainstream statistical theory. It is not a statistical-methodscookbook, nor a compendium of statistical theories, nor is it a mathematics book. The book is intended to be a textbook, aimed for use in the traditional full-year upper-division undergraduate course in probability and statistics, or for use as a text in a course designed for first-year graduate students. The latter course is often a "service course," offered to a variety of disciplines. No previous course in probability or statistics is needed in order to study the book. The mathematical preparation required is the conventional full-year calculus course which includes series expansion, mUltiple integration, and partial differentiation. Linear algebra is not required. An attempt has been made to talk to the reader. Also, we have retained the approach of presenting the theory with some connection to practical problems. The book is not mathematically rigorous. Proofs, and even exact statements of results, are often not given. Instead, we have tried to impart a "feel" for the theory. The book is designed to be used in either the quarter system or the semester system. In a quarter system, Chaps. I through V could be covered in the first

xiv

PREFACE TO THE THIRD EDITION

quarter, Chaps. VI through part of VIII the second quarter, and the rest of the book the third quarter. In a semester system, Chaps. I through VI could be covered the first semester and the remaining chapters the second semester. Chapter VI is a " bridging" chapter; it can be considered to be a part of" probability" or a part of" statistics." Several sections or subsections can be omitted without disrupting the continuity of presentation. For example, any of the following could be omitted: Subsec. 4.5 of Chap. II; Subsecs., 2.6, 3.5, 4.2, and 4.3 of Chap. III; Subsec. 5.3 of Chap. VI; Subsecs. 2.3, 3.4, 4.3 and Secs. 6 through 9 of Chap. VII; Secs. 5 and 6 of Chap. VIII; Secs. 6 and 7 of Chap. IX; and all or part of Chaps. X and XI. Subsection 5.3 of Chap VI on extreme-value theory is somewhat more difficult than the rest of that chapter. In Chap. VII, Subsec. 7.1 on Bayes estimation can be taught without Subsec. 3.4 on loss and risk functions but Subsec. 7.2 cannot. Parts of Sec. 8 of Chap. VII utilize matrix notation. The many problems are intended to be essential for learning the material in the book. Some of the more difficult problems have been starred. ALEXANDER M. MOOD FRANKLIN A. GRAYBILL DUANE C. BOES

EXCERPTS FROM THE FIRST AND SECOND EDITION PREFACES

This book developed from a set of notes which I prepared in 1945. At that time there was no modern text available specifically designed for beginning students of matpematical statistics. Since then the situation has been relieved considerably, and had I known in advance what books were in the making it is likely that I should not have embarked on this volume. However, it seemed sufficiently different from other presentations to give prospective teachers and students a useful alternative choice. The aforementioned notes were used as text material for three years at Iowa State College in a course offered to senior and first-year graduate students. The only prerequisite for the course was one year of calculus, and this requirement indicates the level of the book. (The calculus class at Iowa State met four hours per week and included good coverage of Taylor series, partial differentiation, and multiple integration.) No previous knowledge of statistics is assumed. This is a statistics book, not a mathematics book, as any mathematician will readily see. Little mathematical rigor is to be found in the derivations simply because it would be boring and largely a waste of time at this level. Of course rigorous thinking is quite essential to goocf statistics, and I have been at some pains to make a show of rigor and to instill an appreciation for rigor by pointing out various pitfalls of loose arguments.

XVI

EXCERPTS FROM THE FIRST AND SECOND EDITION PREFACES

While this text is primarily concerned with the theory of statistics, full cognizance has been taken of those students who fear that a moment may be wasted in mathematical frivolity. All new subjects are supplied with a little scenery from practical affairs, and, more important, a serious effort has been made in the problems to illustrate the variety of ways in which the theory may be applied. The probl~ms are an essential part of the book. They range from simple numerical examples to theorems needed in subsequent chapters. They include important subjects which could easily take precedence over material in the text; the relegation of subjects to problems was based rather on the feasibility of such a procedure than on the priority of the subject. For example, the matter of correlation is dealt with almost entirely in the problems. It seemed to me inefficient to cover multivariate situations twice in detail, i.e., with the regression model and with the correlation model. The emphasis in the text proper is on the more general regression model. The author of a textbook is indebted to practically everyone who has touched the field, and I here bow to all statisticians. However, in giving credit to contributors one must draw the line somewhere, and I have simplified matters by drawing it very high; only the most eminent contributors are mentioned in the book. I am indebted to Catherine Thompson and Maxine Merrington, and to E. S. Pearson, editor of Biometrika, for permission to include Tables III and V, which are abridged versions of tables published in Biometrika. I am also indebted to Professors R. A. Fisher and Frank Yates, and to Messrs. Oliver and Boyd, Ltd., Edinburgh, for permission to reprint Table IV from their book " Statistical Tables for Use in Biological, Agricultural and Medical Research." Since the first edition of this book was published in 1950 many new statistical techniques have been made available and many techniques that were only in the domain of the mathematical statistician are now useful and demanded by the applied statistician. To include some of this material we have had to eliminate other material, else the book would have come to resemble a compendium. The general approach of presenting the theory with some connection to practical problems apparently contributed significantly to the success of the first edition and we have tried to maintain that feature in the present edition.

I
PROBABILITY

1 INTRODUCTION AND SUMMARY

The purpose of this chapter is to define probability and discuss some of its properties. Section 2 is a brief essay on some of the different meanings that have been attached to probability and may be omitted by those who are interested only in mathematical (axiomatic) probability, which is defined in Sec. 3 and used throughout the remainder of the text. Section 3 is subdivided into six subsections. The first, Subsec. 3.1, discusses the concept of probability models. It provides a real-world setting for the eventual mathematical definition of probability. A review of some of the set theoretical concepts that are relevant to probability is given in Subsec. 3.2. Sample space and event space are defined in Subsec. 3.3. Subsection 3.4 commences with a recall of the definition of a function. Such a definition is useful since many of the words to be defined in this and coming chapters (e.g., probability, random variable, distribution, etc.) are defined as particular functions. The indicator function, to be used extensively in later chapters, is defined here. The probability axioms are presented, and the probability function is defined. Several properties of this probability function are stated. The culmination of this subsection is the definition of a probability space. Subsection 3.5 is devoted to examples of probabilities

PROBABIUTY

defined on finite sample spaces. The related concepts of independence of events and conditional probability are discussed in the sixth and final subsection. Bayes' theorem, the mUltiplication rule, and the theorem of total probabilities are proved or derived, and examples of each are given. Of the three main sections included in this chapter, only Sec. 3, which is by far the longest, is vital. The definitions of probability, Pfobabiijly space, conditional probability, and independence, along with familiarity with the properties of probability, conditional and unconditional and related formulas, are the essence of this chapter. This chapter is a background chapter; -it introduces the language of probability to be used in developing distribution theory, which is the backbone of the theory of statistics.

2 2.1

KINDS OF PROBABILITY Introduction

One of the fundamental tools of statistics is probability, which had its formal beginnings with games of chance in the seventeenth century. Games of chance, as the name implies, include such actions as spinning a roulette wheel, throwing dice, tossing a coin, drawing a card, etc., in which th~ outcome of a trial is uncertain. However, it is recognized that even though the outcome of any particular trial may be uncertain, there is a predictable' ibngterm outcome. It is known, for example, that in many throws of an ideal (balanced, symmetrical) coin about one-half of the trials will result in heads. It is this long-term, predictable regularity that enables gaming houses to engage in the business. A similar type of uncertainty and long-term regularity often occurs in experimental science. For example, in the science of genetics it is uncertain whether an offspring will be male or female, but in the long run it is known approximately what percent of offspring will be male and what percen't will be female. A life insurance company cannot predict which persons in the United States will die at age 50, but it can predict quite satisfactorily how many people in the United States will die at that age. ~ First we shall discuss the classical, or a priori, theory of probability; then we shall discuss the frequency theory. Development of the axiomatic approach will be deferred until Sec. 3.

KINDS OF PROBAMUl'Y

2.2

Classical or A Priori Probability

As we stated in the previous subsection, the theory of probability in its early stages was closely associated with games of chance. This association prompted the classical definition. For example, suppose that we want the probability of the even.t1hat an ideal coin will turn up heads. We argue in this manner: Since there are only two ways that the coin can fall, heads or tails, and since the coin is well balanced, one would expect that the coin is just as likely to fall heads as tails; h~nce, the probability of the event of a head will be given the value t This kind of reasoning prompted the following classical definition of probability.
_
~

>-a(

Definition 1 Classical probability If a random experiment can resul t in n mutually exclusive and equally likely outcomes and if nA. of these outcomes have an attribute A, then the probability of A is the fraction nA./n. 1/1/
We shall apply this definition to a few examples in order to illustrate its meaning. If an ordinary die (one ofa pair of dice) is tossed-there are six possible outcomes-anyone of the six numbered faces may turn up. These six outcomes are mutually exclusive since two or more faces cannot turn up simultaneously. And if the die is fair, or true, the six outcomes are equally likely; i.e., it is expected that each face will appear with about equal relative frequency in the long run. blow suppose that we want the probability that the result of a toss be an even number. Three of the six possible outcomes have this attribute. The probability t~at an even number will appear when a die is tossed is therefore i, or t. -l;imilarly, the probability that a 5 will appear when a die is tossed is 1. The probability that the result of a toss will be greater than 2 is iTo consider another example, suppose that a card is drawn at random from an ordinary deck of playing cards. The probability of drawing a spade is readily seen to be ~ ~, or i The probability of drawing a number between 5 and lO~ inclusive, is ; ~, or t'3 The application of the definition is straightforward enough in these simple cases, but it is not always so obvious. Careful attention must be paid to the q~<!1ifications "mutually exclusive," equally likely," and random." Suppose that one wishes to compute the probability of getting two heads if a coin is tossed twice. He might reason that there are three possible outcomes for the two tosses: two "heads, two tails, or one head and one tai 1. One of these three
H H

.
4
PROBABIUTY'

outcomes has the desired attribute, i.e., two heads; therefore the probability is -1. This reasoning is faulty because the three given outcomes are not equally likely. The third outcome, one head and one tail, can OCCUr in two ways since the head may appear on the first toss and the tail on the second or the head may appear on the second toss and the tail on the first. Thus there are four equally likely outcomes: HH, HT, TH, and TT. The first of these has the desired attribute, while the others do not. The correct probability is therefore i. The result would be the same if two ideal coins were tossed simultaneously. Again, suppose that one wished to compute the probability that a card drawn from an ordinary well-shuffied deck will be an ace or a spade. In enumerating the favorable outcomes, one might count 4 aces and 13 spades and reason that there are 17 outcomes with the desired attribute. This is clearly incorrect because these 17 outcomes are not mutually exclusive since the ace of spades is both an ace and a spade. There are 16 outcomes that are favorable to an ace or a spade, and so the correct probability is ~ ~, or 143' We note that by the classical definition the probability of event A is a number between 0 and 1 inclusive. The ratio nAln must be less than or equal to 1 since the total number of possible outcomes cannot be smaller than the number of outcomes with a specified attribute. If an event is certain to happen, its probability is 1; if it is certain not to happen, its probability is O. Thus, the probability of obtaining an 8 in tossing a die is O. The probability that the number showing when a die is tossed is less than 10 is equal to 1. The probabilities determined by the classical definition are called a priori probabilities. When one states that the probability of obtaining a head in tossing a coin is !-, he has arrived at this result purely by deductive reasoning. The result does not require that any coin be tossed or even be at hand. We say that if the coin is true, the probability of a head is !, but this is little more than saying the same thing in two different ways. Nothing is said about how one can determine whether or not a particular coin is true. The fact that we shall deal with ideal objects in developing a theory of probability will not troub1e us because that is a common requirement of mathematical systems. Geometry, for example, deals with conceptually perfect circles, lines with zero width, and so forth, but it is a useful branch of knowledge, which can be applied to diverse practical problems. There are some rather troublesome limitations in the classical, or a priori, approach. It is obvious, for example, that the definition of probability must be modified somehow when the total number of possible outcomes is infinite. One might seek, for example, the probability that an integer drawn at random from the positive integers be even. The intuitive answer to this question is t.

KINDS OF PROBABIUTY

If one were pressed to justify this result on the basis of the definition, he might reason as fol1ows: Suppose that we limit ourselves to the first 20 integers; 10 of these are even so that the ratio of favorable outcomes to the total number is t%, or 1 Again, if the first 200 integers are considered, 100 of these are even, and the ra~io is also 1_ In genera!, the first 2N integers contain N even integers; if we form the ratio N/2N and let N become infinite so as to encompass the whole set of positive integers, the ratio remains 1_ The above argument is plausible, and the answer is plausible, but it is no simple matter to make the argument stand up. It depends, for example, on the natural ordering of the positive integers, and a different ordering could produce a different result. Thus, one could just as well order the integers in this way: 1,3,2; 5, 7,4; 9, 11,6; ___ , taking the first pair of odd integers then the first even integer, the second pair of odd integers then the second even integer, and so forth. With this ordering, one could argue that the probability of drawing an even integer is 1-. The integers can also be ordered so that the ratio will oscillate and never approach any definite value as N increases. There is another difficulty with the classical approach to the theory of probability which is deeper even than that arising in the case of an infinite num ber of outcomes. Suppose that we toss a coin known to be biased in favor of heads (it is bent so that a head is more likely to appear than a tail). The two possible outcomes of tossing the coin are not equally likely. What is the probability of a head? The classical definition leaves us completely helpless here. Still another difficulty with the classical approach is encountered when we try to answer questions such as the following: What is the probability that a child born in Chicago will be a boy? Or what is the probability that a male will die before age 50? Or what is the probability that a cookie bought at a certain bakery will have less than three peanuts in it? All these are legitimate questionswhichwewantto bring into the realm of probability theory. However, notions of symmetry," "equally likely," etc., cannot be utilized as they could be in games of chance. Thus we shall have to alter or extend our definition to bring problems similar to the above into the framework of the theory. This more widely applicable probability is called a posteriori probability, or!requency, and will be discussed in the next subsection.
H

2.3

A Posteriori or Frequency Probability

A coin which seemed to be well balanced and symmetrical was tossed 100 times, and the outcomes recorded in Table 1. The important thing to notice is that the relative frequency of heads is close to 1. This is not unexpected since the coin

PROBABILITY

was symmetrical, and it was anticipated that in the long run heads would occur about one-half of the time. For another example, a single die was thrown 300 times, and the outcomes recorded in Table 2. Notice how close the relative frequency of a face with a I showing is to !; similarly for a 2, 3, 4, 5, and 6. These results are not unexpected since the die which was used was quite symmetrical and balanced; it was expected that each face would occur with about equal frequency in the long run. This suggests that we might be willing to use this relative frequency in Table 1 as an approximation for the probability that the particular coin used will come up heads or we might be willing to use the relative frequencies in Table 2 as approximations for the probabilities that various numbers on this die will appear. Note that although the relative frequencies of the different outcomes are predictable, the actual outcome of an individual throw is unpredictable. In fact, it seems reasonable to assume for the coin experiment that there exists a number, label it p, which is the probability of a head. Now if the coin appears well balanced, symmetrical, and true, we might use Definition 1 and state that p is approximately equal to 1. It is only an approximation to set p equal to 1- since for this particular coin we cannot be certain that the two cases, .heads and tails, are exactly equally likely. But by examining the balance and symmetry of the coin it may seem quite reasonable to assume that they are. Alternatively, the coin could be tossed a large number of times, the results recorded as in Table 1, and the relative frequency ofa head used as an approximation for p. In the experiment with a die, the probability P2 of a 2 showing could be approximated by using Definition I or by using the relative frequency in Table 2. The important thing is that we postulate that there is a number P which is defined as the probability of a head with the coin or a number P2 which is the probability of a 2 showing in the throw of the die. Whether we use Definition 1 or the relative frequency for the probability seems unimportant in the examples ci ted.

Table 1 RESULTS OF TOSSING A COIN 100 TIMES Long-run expec~ed relative frequency of a balanced coin
.50 .50 1.00

Outcome
H T

Observed Frequency
56 44 100

Observed relative frequency

.56 .44 1.00

Total

KINDS OF PROBABIUTY

Suppose, as described above, that the coin is unbalanced so that we are quite certain from an examination that the two cases, heads and tails, are not equally likely ,to happen. In these cases a number p can still be postulated as the probabilit y that a head shows, but the classical definition will not help us to find the value of p. We must use the frequency approach or possibly some physical arialysis of the unbalanced coin. In many scientific investigations, observations are taken which have an element of uncertainty or unpredictability in them. As a very simple example, suppose that we want to predict whether the next baby born in a certain locality will be a male or a female. This is individually an uncertain event, but the results of groups of births can be dealt with satisfactorily. We find that a certain longrun regularity exists which is similar to the long-run regularity of the frequency ratio of a head when a coin is thrown. If, for example, we find upon examination of records that about 51 percent of the births are male, it might be reasonable to postulate that the probability of a male birth in this locality is equal to a number p and take .51 as its approximation. To make this idea more concrete, we sha11 assume that a series of observations (or experiments) can be made under quite uniforlJ1 conditions. That is, an observation of a random experiment is made; then the experiment is repeated under similar conditions, and another observation taken. This is repeated many times, and while the conditions are similar each time, there is an uncontrollable variation which is haphazard or random so that the observations are individually unpredictable. In many of these cases the observations fall into certain classes wherein the relative frequencies are quite stable. This s~ggests that we postulate a number p, called the probability of the event, and approximate p by the ~elative frequency with which the repeated observations satisfy the

Table 2 RESULTS OF TOSSING A DIE 300 TIMES

Outcome 1

Observed Frequency
51

Observed relative frequency .170 .180 .160 .170 .163 .157

1.000

Long-run expected relative frequency of a balanced die

.1667 .1667 .1667 .1667 .1667 .1667 1.000

2
3 4 5 6

54 48 51 49 47
300

Tota1

PROBABILITY

event. For instance, suppose that the experiment consists of sampling the population of a large city to see how many voters favor a certain proposal. The outcomes are" favor" or "do not favor," and each voter's response is unpredictable, but it is reasonable to postulate a number p as the probability that a given response will be "favor." The relative frequency of" favor" responses can be used as an approximate value for p. As another example, suppose that the experiment consists of sampling transistors from a large collection of transistors. We shall postulate that the probability of a given transistor being defective is p. We can approximate p by selecting several transistors at random from the collection and computing the relative frequency of the number defective. The important thing is that we can conceive of a series of observations or experiments under rather uniform conditions. Then a number p can be postulated as the probability of the event A happening, and p can be approximated by the relative frequency of the event A in a series of experiments.

3 PROBABILITY-AXIOMATIC
3.1 Probability Models

One of the aims of science is to predict and describe events in the world in which we live. One way in which this is done is to construct mathematical models which adequately describe the real world. For example, the equation s = tgt 2 expresses a certain relationship between the symbols s, g, and t. It is a mathematical model. To use the equation s = !gt 2 to predict s, the distance a body falls, as a function of time t, the gravitational constant g must be known. The latter is a physical constant which must be measured by experimentation if the equation s = tgt 2 is to be usefuL The reason for mentioning this equation is that we do a similar thing in probability theory; we construct a probability model which can be used to describe events in the real world. For example, it might be desirable to find an equation which could be used to predict the Sex of each birth in a certain locality. Such an equation would be very complex, and none has been found. However, a probability model can be constructed which, while not very helpful in dealing with an individual birth, is quite useful in dealing with groups of births. Therefore, we can postulate a number p which represents the probability that a birth will be a male. From this fundamental probability we can answer questions such as: What is the probability that in ten births at least three will be males? Or what is the probability that there will be three consecutive male births in the next five? To answer questions such as these and many similar ones, we shall develop an idealized probability model.

PROBABILlTY-AXIOMATIC

The two general types of probability (a priori and a posteriori) defined above have one important thing in common: They both requir~ a conceptual experiment in which the various outcomes can occur under somewhat uniform conditions. For example, repeated tossing of a coin for the a priori case, and repeated birth for the a posteriori case. However, we might like to bring into the realm of probability theory situations which cannot conceivably fit into the framework of repeated outcomes under somewhat similar conditions. For example, we might like to answer questions such as: What is the probability my wife loves me? Or what is the probability that World War III will start before January 1, 1985? These types of problems are certainly a legitimate part of general probability theory and are included in what is referred to as subjective probability. We shall not discuss subjective probability to any great extent in this book, b~t we remark that the axioms of probability from which we develop probability theory are rich enough to include a priori probability, a posteriori probability, and subjective probability. To start, we require that every possible outcome of the experiment under study can be enumerated. For example, in the coin-tossing experiment there are two possible outcomes: heads and tails. We shall associate probabilities only with these outcomes or with collections of these outcomes. We add, however, that even if a particular outcome is impossible, it can be included (its probability is 0). The main thing to remember is that every outcome which can occur must be included. Each conceivable outcome of the conceptual experiment under study will be defined as a sample point, and the totality of conceivable outcomes (or sample points) will be defined as the sample space. Our .object, of course, is to assess the probability of certain outcomes or collections of outcomes of the experiment. Discussion of such probabilities is conveniently couched in the language of set theory, an outline of which appears in the next subsection. We shaH return to formal definitions and examples of sample space, event, and probability.

3.2

An Aside-Set Theory

We begin with a collection of objects. Each object in our collection will be called a point or element. We assume that our collection of objects is large enough to include all the points under consideration in a given discussion. The totality of all these points is called the space, universe, or universal set. We will call it the space (anticipating that it will become the sample space when we speak of probability) and denote it by n. Let (1) denote an element or point i"n n. Although a set can be defined as any collection of objects, we shall

10

PROBABIliTY

assume, unless otherwise stated, that all the sets mentioned in a given discussion consist of points in the space n. EXAMPLE 1 Q = R 2 , where R2 is the collection of points co in the plane and co = (x, y) is any pair of real numbers x and y. IIII EXAMPLE 2

n = {all United States citizens}.

IIII

We shall usually use capital Latin letters from the beginning of the alphabet, with or without subscripts, to denote sets. If co is a point or element belonging to the set A, we shall write co E A; if co is not an element of A, we shall write co ; A.

Definition 2 Subset If every element of a set A is also an element of a set B, then A is defined to be a subset of B, and we shall write A c B or B => A; read" A is contained in B" or " B contains A." II1/ Definition 3 Equivalent sets Two sets A and B are defined to be equivalent, or equal, if A c Band B c A.. This will be indicated by writing
A =B.

/111
\'vill

Definition 4 Empty set If a set A contains no points, it the null set, or empty set, and denoted by </>.

be called

/1//

Definition S Complement The complement of a set A with respect to the space n, denoted by A, A or n - A, is the set of all points that are in Q but not in A. 1//1
C ,

Definition 6 Union Let A and B be any two subsets of n; then the set that consists of all points that are in A or B or both is defined to be the union of A and B and written A vB. 1/// Definition 7 Intersection Let A and B be any two subsets of n; then the set that consists of all points that are in both A and B is defined to be / / // the intersection of A and B and is written A (\ B or AB. Definition 8 Set difference Let A and B be any two subsets of n. The set of all points in A that are not in B wHl be denoted by A - B and is II// defined as set difference.

PROBABILITY-AXIOMATIC

11

EXAMPLE 3 Let n = {(x, y): 0 <x ~ 1 and 0 <y< I}, which is read the collection of all points (x, y) for which 0 < x < 1 and 0 < y < 1. Define the following sets: Ai = {(x, y): 0 < x < 1; 0 < Y <
A2 A3 = {(x, y): 0 < x < y < l}, A4

t},

= {(x, y): 0 < x <!; 0 <y< I}, = {(x, y): 0 <x<!; 0 <y< !}.

(We shall adhere to the practice initiated here of using braces to embrace the points of a set.) The set relations below follow.
A1 (\ A2 = A1A2 = A4; A1 ={(x,y):O<x< l;!<y< I}; Al - A4
=

{(x, y):

!<x<

I; 0 < Y < !}.

IIII

EXAMPLE 4 Let n, A 1, A 2 , and A3 be as indicated in the diagrams in Fig. I IIII which are called Venn diagrams. The set operations of complement, union, and intersection have been defined in Definitions 5 to 7, respectively. These set operations satisfy quite a number of laws, some of which follow, stated as theorems. Proofs are omitted.

Theorem 1 Commutative laws A u B

B u A and A (\ B = B (\ A.

IIII
Theorem 2 Associative laws A u (B u C) = (A u B) u C, and
A (\ (B (\ C) = (A (\ B) (\ C.

IIII

Theorem 3 Distributive laws A (\ (B u C) = (A (\ B) u (A (\ C), and A u (B (\ C) = (A u B) (\ (A.u C). IIII Theorem 4 (Ac)C = (1) = A; in words, the complement of A complement equals A.

IIII IIII IIII

Theorem 5 An = A; A u n = n; A4> = 4>; and A u 4> = A. Theorem 6

AA
=

4>; A u A = n; A (\ A = A; and A u A

= A.

12 PROBABILITY

n is the entire square

FIGURE 1

PROBABILITY-AXIOMATIC

13

Theorem 7 (A u B) = A n 13, and (A n B) = A u 13. These are known as De Morgan's laws. IIII Theorem 8 A - B
=

AB.

Jill

Several of the above laws are illustrated in the Venn diagrams in Fig. 1. Although we will feel free to use any of the above laws, it might be instructive to give a proof of one of them just to illustrate the technique. For example, let us show that (A u B) = A n B. By definition, two sets are equal if each is contained in the other. We first show that (A u B) cAn 13 by proving that if OJ E A u B, then OJ E A n 13. Now OJ E (A u B) implies OJ A u B, which implies that OJ A and OJ B, which in turn implies that OJ E A and OJ E 13; that is, OJ E A n 13. We next show that A nBc (A u B). Let OJ E A n 13, which means OJ belongs to both A and 13. Then OJ A u B for if it did, OJ must belong to at least one of A or B, contradicting that OJ belongs to both A and 13; however, OJ A u B means OJ E (A u B), completing the proof. We defined union and intersection of two sets; these definitions extend immediately to more than two sets, in fact to an arbitrary number of sets. It is customary to distinguish between the sets in a collection of subsets of n by assigning names to them in the form of subscripts. Let A (Greek letter capital lambda) denote a catalog of names, or indices. A is also called an index set. For example, if we are concerned with only two sets, then our index set A includes only two indices, say I and 2; so A = {I , 2}.

Definition 9 Union and intersection of sets Let A be an index set and {AA,= XE A} = {AA,}' a collection of subsets of n indexed by A. The set of points that consists of all points that belong to AA, for at least one A is called the union of the sets {AA,} and is denoted by UAA,' The set of points
A.
E

that consists of all points that belong to A A, for every A is called the interAA,' If A is empty, then define section of the sets {AA,} and is denoted by
A.EA

UAA, = ljJ and

A.EA

nAA, = n.
A.eA

A.

n
E

IIII

EXAMPLE 5 If A = {1, 2, ... , N}, i.e., A is the index set consisting of the AA, is also written as first N integers, then

n=l

U An = A 1 U

A2

U ... U

AN'

IIII

14

PROBABILlTY

One of the most fundamental theorems relating unions, intersections, and complements for an arbitrary collection of sets is due to De Morgan. Theorem 9 De Morgan's theorem Let A be an index set and {A;J a collection of subsets of il indexed by A. Then,

IIII
We wi1l not give a proof of this theorem. Note, however, that the special case when the index set A consists of only two names or indices is Theorem 7 above, and a proof of part of Theorem 7 was given in the paragraph after Theorem 8. Definition 10 Disjoint or mutually exclusive Subsets A and B of Q are defined to be mutually exclusive or disjoint if A (') B = l/J. Subsets AI, A 2 , are defined to be mutually exclusive if AjAj = l/J for every i =# j.

IIII
Theorem 10 If A and B are subsets of (ii) AB (') AB = l/J.
(i) A = A () Q =AABB=Al/J = l/J.
PROOF

Q,

then (i) A

ABu AB, and

= A () (B u B) =

AB u AB.

(ii) AB (') AB

M
IIII

Theorem 11
PROOF

If A c B, then AB = A, and A u B = B. Left as an exercise.

3.3

Definitions of Sample Space and Event

In Subsec. 3.1 we described what might be meant by a probability model. There we said that we had in mind some conceptual experiment whose possible outcomes we would like to study by assessing the probability of certain outcomes or collection of outcomes. In this subsection, we win give two important definitions, along with some examples, that will be used in assessing these probabilities. Definition 11 Sample space The sample space, denoted by il, is the collection or totality of all possible outcomes of a conceptual experiment.

IIII

PROBABILITY-AXIOMATIC

15

One might try to understand the definition by looking at the individual words. Use of the word "space" can be justified since the sample space is the total collection of objects or elements which are the outcomes of the experiment. This is in keeping with our use of the word U space" in set theory as the collection of all objects of interest in a given discussion. The word ~'sample" is harder to justify; our experiment is random, meaning that its outcome is uncertain so that a given outcome is just one sample of many possible outcomes. Some other symbols that are used in other texts to denote a sample space, in addition to Q, are S, Z, R, E, X, and A. Definition 12 Event and event space An event is a subset of the sample space. The class of all events associated with a given experiment is defined to be the event space. 1//1 The above does not precisely define what an event is. An event will always be a subset of the sample space, but for sufficiently large sample spaces not all subsets will be events. Thus the class of all subsets of the sample space will not necessarily correspond to the event space. However, we shall see that the class of all events can always be selected to be large enough so as to include all those subsets (events) whose probability we may want to talk about. If the sample space consists of only a finite number of points, then the corresponding event space will be the class of all subsets of the sample space. Our primary interest will not be in events per se but will be in the probability that an event does or does not occur or happen. An event A is said to occur if the experiment at hand results in an outcome (a point in our sample space) that belongs to A. Since a point, say OJ, in the sample space is a subset (that subset consisting of the point OJ) of the sample space n, it is a candidate to be an event. Thus OJ can be viewed as a point in n or as a subset of n. To distinguish, let us write {OJ}, rather than just OJ, whenever (J) is to be viewed as a subset of n. Such a one-point subset will always be an event and will be called an elementary event. Also if> and n are both subsets of Q, and both will always be events. n is so metimes called the sure event. We shall attempt to use only capital Latin letters (usually from the beginning of the alphabet), with or without affixes, to denote events, with the exception that if> will be used to denote the empty set and n the sure event. The event space will always be denoted by a script Latin letter, and usually d. ;?4 and /F, as well as other symbols, are used in some texts to denote the class of all events. The sample space is basic and generally easy to define for a given experiment. Yet, as we shall see, it is the event space that is really essential in defining probabiHty. Some examples follows.

16

PROBABILITY

EXAMPLE 6 The experiment is the tossing of a single die (a regular sjx-sided polyhedron or cube marked on each face with one to six spots) and noting which face is up. Now the die can land with anyone of the six faces up; so there are six possible outcomes of the experiment;

n = { [:J,

G:J, I.', 1::1, (:-:" nil}.

Let A = {even number of spots up}. A is an event; it is a subset of n. A = {0, @], [ll)}. Let Ai = {i spots up}; i = ], 2, ... , 6. Each Ai is an elementary event. For this experiment the sample space is finite; hence the event space is all subsets of n. There are 26 = 64 events, of which only 6 are elementary, in J7I (including both q, and n). See Example 19 of Subsec. 3.5, where a technique for counting the number of events in a finite 1/// sample space is presented.

EXAMPLE 7 Toss a penny, nickel, and dime simultaneously, and note which side is up on each. There are eight possible outcomes of this experiment. n ={(H, H, H), (H, H, T), (H, T, H), (T, H, H), (H, T, (T, H, (T, T, H), (T, T, T)}. We are using the first position of (', " .), called a 3-tuple, to record the outcome of the penny, the second position to record the outcome of the nickel, and the third position to record the outcome ot the dime. Let Ai = {exactly i heads}; i = 0, I, 2, 3. For each i, Ai is an event. Note that Ao and A3 are each elementary events. Again all subsets of n are events; there are 2 8 = 256 of them. IIII

n,

n,

EXAMPLE 8 The experiment is to record the number of traffic deaths in the state of Colorado next year. Any nonnegative integer is a conceivable outcome of this experiment; so n = {O, I, 2, ... }. A = {fewer than 500 deaths} = {O, I, ... , 499} is an event. Ai = {exactly i deaths}, i = 0, 1, ... , is an elementary event. There is an infinite number of points in the sample space, and each point is itself an (elementary) event; so there is an infinite number of events. Each subset of n is an event. III/

EXAMPLE 9 Select a light bulb, and record the time in hours that it burns before burning out. Any nonnegative number is a conceivable outcome of this experiment; so n = {x: x >O}. For this sample space not an

PROBABILITY-AXIOMATIC

17

subsets of Q are events; however, any subset that can be exhibited will be an event. For example, let
A = {bulb burns for at least k hours but burns out before m

hours}
= {x:

k < x < m};

then A is an event for any 0 < k < m.

II1I

EXAMPLE J0 Consider a random experiment which consists of counting the number of times that it rains and recording in inches the total rainfall next July in Fort Collins, Colorado. The sample space could then be represented by
Q

= {(I,

x): i

= 0, 1, 2, . .. and 0

x},

where in the 2-tuple (', .) the first position indicates the number of times that it rains and the second position indicates the total rainfall. For example, OJ = (7, 2.251) is a point in Q corresponding to there being seven different times that it rained with a total rainfall of 2.251 inches. A = {(i, x): i = 5, ... , 10 and x 3} is an example of an event. IIII

EXAMPLE 11 In an agricultural experiment, the yield of five varieties of wheat is examined. The five varieties are all grown under rather uniform conditions. The outcome is a collection of five numbers (Yl ,Y2 ,Y3 'Y4 ,Ys), where Yi represents the yield of the ith variety in bushels per acre. Each Yi can conceivably be any real number greater than or equal to O. In this example let the event A be defined by the conditions that Y2, Y3' Y4' and Ys are each 10 or more bushels per acre larger than Yl, the standard variety. In our notation we write

Our definition of sample space is precise and satisfactory, whereas our definitions of event and event space are not entirely satisfactory. We said that if the sample space was" sufficiently large" (as in Examples 9 to 11 above), not all subsets of the sample space would be events; however, we did not say exactly which subsets would be events and which would not. Rather than developing the necessary mathematics to precisely define which subsets of Q constitute OUf

18

PROBABILITY

event space d, let us state some properties of d that it seems reasonable to . requIre: (i)
(ii) (iii)
QEd.

If A

d, then

A E d.
U

If Al and A2 Ed, then Al

A2 Ed.

We said earlier that we were interested in events mainly because we would be interested in the probability that an event happens. Surely, then, we would want d to include Q, the sure event. Also, if A is an event, meaning we can talk about the probability that A occurs, then A should also be an event so that we can talk about the probability that A does not occur. Similarly, if Al and A2 are events, so should Al U A2 be an event. Any collection of events with properties (i) to (iii) is caned a Boolean algebra, or just algebra, of events. We might note that the collection of all subsets of Q necessarily satisfies the above properties. Several results follow from the above assumed properties of d.

Theorem 12 t/> E d.
PROOF

By property (i) QEd; by (ii) QEd; but Q = t/>; so t/> Ed.

1111
Theorem 13 If Al and A2 Ed, then Ai () A2 Ed.
PROOF

Al and A2 Ed; hence Al

A2 , and (AI

A 2) Ed, but

(-=A:--1-U-A='2)

= Al () 12 = Al () A2 by De Morgan's law.

1111
n

Theorem 14 If AI' A2 , ... , An Ed, then

PROOF

U Ai and In Ai E d. 1=1 1
n

Follows by induction.

1111

We will always assume that our collection of events d is an algebrawhich partially justifies our use of d as our notation for it. In practice, one might take that collection of events of interest in a given consideration and enlarge the collection, if necessary, to include (i) the sure event, (ii) all complements of events already included, and (iii) all finite unions and intersections of events already included, and thIS will be an algebra d. Thus far, we have not explained why d cannot always be taken to be the collection of all subsets of Q. Such explanation will be given when we define probability in the next subsection.

P:aOBABILITY-AXIOMATIC

19

3.4

Definition of Probability

In this section we give the axiomatic definition of probability. Although this formal definition of probability will not in itself allow us to achieve our goal :>f assigning actual probabilities to events consisting of certain outcomes of random experiments, it is another in a series of definitions that will ultimately lead to that goal. Since probability, as well as forthcoming concepts, is defined as a particular function, we begin this subsection with a review of the notion of a function. The following terminology is frequently used to describe a function: A function, say f('), is a rule (law, formula, recipe) that associates each point in one set of points with one and only one point in another set of points. The first collection of points, say A, is caned the domain, and the second collection, say B, the counterdomain.

The definition of a function

Definition 13 Function A function, say f( .), with domain A and counterdomain B, is a collection of ordered pairs, say (a, b), satisfying (i) a E A and b E B; tii) each a E A occurs as the first element of some ordered pair in the collection (each bE B is not necessarily the second element of some ordered pair); and (iii) no two (distinct) ordered pairs in the collection IIII have the same first element. If (a, b) Ef( .), we write b = f(a) (read" b equals f of a") and call f(a) the value of f() at a. For any a E A, f(a) is an element of B; whereasf( -) is a set of Qrdered pairs. The set of all values of f( .) is called the range of f( ); i.e., the range of f(') = {b E B: b = f(a) for some a E A} and is always a subset of the counterdomain B but is not necessarily equal to it. f(a) is also called the image of a under f( +), and a is called the pre image of /(a).

EXAMPLE 12 Let.ft(-) andf2(') be the two functions, having the real line for their domain and counterdomain, defined by

fi ( .) = {(x, y) : y = x 3 + X + 1, and f2(' )

00

< x < oo}

= {(x, y): y = x 2, -

00

< x < oo}.

The range offi ( .) is the counterdomain, the whole real line, but the range of f2(') is all nonnegative real numbers, not the same as the counterdomain_ Jill

20 . PROBABILITY

Of particular interest to us will be a class of functions that are called indicator functions.

Definition 14 Indicator function Let

Q be any space with points

and A any subset of Q. The indicator function of A, denoted by [A('), is the function with domain Q and counterdomain equal to the set consisting of the two real numbers 0 and 1 defined by if if
[A(') clearly
U

WE
W

A A.

indicates" the set A.

IIII

Properties of Indicator Functions Let Q be any space and d any collection

of subsets of Q:
(i)

[A(W) = 1 - [A(W) for every A

d.
E

(ii) (iii)

IAIA2'''A,,(W) = IAt(w)' I A2(w)'" IA,,(w) for'Aj, ... , An

d.

I Al VA2V ... VAn(W) = max [lAt( w), I A2( w), .. , I An(W)] for AI' ... , An E d.
[~(w)

(iv)

[A(W) for every A

d.

Proofs of the above properties are left as an exercise. The indicator function will be used to "indicate" subsets of the real line; e.g.,
I([o.I))(X)

= lto lx) =

{~

if 0 <x < 1 otherwise,

and if [+ denotes the set of positive integers, if x is some positive integer otherwise. Frequent use of indicator functions will be made throughout the remainder of this book. Often the utility of the indicator function is just notational efficiency as the following example shows.

EXAMP LE ] 3 Let the function I( .) be defined by

o
I{x)

x 2-x

for for for for

x<O O<x<l l<x<2 2 < x.

PROBABIUTY-AXIOMATIC

21

By using the indicator function, f(x) can be written as

f(x)

= x/(o, l](x) + (2 f(x) = (1

x)/(l, 2lx ).

or also by using the absolute value symbol as

-II -

x 1)/(0. 2](X).

IIII

Another type of function that we will.have occasion to discuss is the set function defined as any functon which has as its domain a collection of sets and as its counterdomain the real line including, possibly, infinity. Examples of set functions follow.

EXAMPLE 14 Let 0 be the sample space corresponding to the experiment of tossing two dice, and let d be the collection of all subsets of O. For any A E d define N(A) = number of outcomes, or points in 0, that are in A. Then N( <p) = 0, N(O) = 36, and N(A) = 6 if A is the event containing those outcomes having a total of seven spots up. If1/

The size-oJ-set function alluded to in the above example can be defined. in general, for any set A as the number of points in A, where A is a member of an arbitrary collection of sets d.

EXAMPLE 15 Let 0 be the plane or two-dimensional euclidean space and d any collection. of subsets of 0 for which area is meaningful. Then for any A Ed define Q(A) = area of A. For example, if A {(x, y); 0 < x <1,0 <y < I}, then Q(A) = 1; if A = {(x, y): x 2 + y2 r2}, then Q(A) = rcr 2 ; and if A = {(O. 0), (1, I)} then Q(A) = O. IIII

The probability function to be defined will be a particular set function.

Probability function Let 0 denote the sample space and d denote a collection of events assumed to be an algebra of events (see Subsec. 3.3) that we shall consider for some random experiment.

22

PROBABILITY

(I

Definition 15 Probability function A probability function P[ ] is a set function with domain d (an algebra of events)* and counterdomain the interval [0, 1] which satisfies the following axioms:
(i) P[A] > 0 for every A Ed.
(ii) prO] = L

(iii)

If AI. A 2
(l

is a sequence of mutually exclusive events in d

U

(that is, Ai

Aj

= 4> for i =1= j; i,j = 1,2, ... ) and if Ai

A2

U ...

IIII
These axioms are certainly motivated by the definitions of classical and frequency probability. This definition of probability is a mathematical definition; it tells us which set functions can be called probability functions; it does not tell us what value the probability function P[] assigns to a given event A. We will have to model our random experiment in some way in order to obtain values for the probability of events. P[A] is read" the probability of event A" or "the probability that event A occurs," which means the probability that any outcome in A occurs. We have used brackets rather than parentheses in our notation for a probability function, and we shall continue to do the same throughout the remainder of this book.
*In defining a probability function, many authors assume that the domain of the set function is a sigma-algebra rather than just an algebra. For an algebra d, we had the property if A1 and A2 E d, then Al U A2 E d. A sigma-algebra differs from an algebra in that the above property is replaced by
00

if A tt A 2 ..... A n ... Ed, then U AnEd.

If'"

It can be shown that a sigma-algebra is an algebra, but not necessarily ronversely.

If the domain of the probability function is taken to be a sigma-algebra then axiom (iii) can be simplified to

if A 1 , A 2

is a sequence of mutually exclusive events in d.

p[ ~ At] = f P[AtJ.
1~1

1=1

A fundamental theorem of probability theory. called the extension theorem. states that if a probability function is defined on an algebra (as we have done) then it can be extended to a sigma-algebra. Since the probability function can be extended from an algebra to a sigma-algebra, it is reasonable to begin by assuming that the probability function is defined on a sigma-algebra.

PROBABILITY-AXlOMATIC

23

EXAMPLE 16 Consider the experiment of tossing two coins, say a penny and a nickel. Let n {(H, H), (H, (T, H), (T, T)} where the first component of (', .) represents the outcome for the peJ}ny. Let us model this random experiment by assuming that the four points in n are equally likely; that is, assume P[{(H, H)}] = P[{~fI, T)}] = P[{(T, H)}] = P[{(T, T)}]. The following question arises: Is the P[ ] function that is implicitly defined by the above really a probability function; that is, does it satisfy the three axioms? It can be shown that it does, and so it is a probability function.

n,

In our definitions of event and .PI, a collection of events, we stated that .PI cannot always be taken to be the collection of all subsets of n. The reason for this is that for" sufficiently large" n the collection of all subsets of n is so large that it is impossible to define a probability function consistent with the above aXIOms. We are able to deduce a number of properties of our function P[ ] from its definition and three axioms. We list these as theorems. I t is in the statements and proofs of these properties that we will see the convenience provided by assuming.PI is an algebra of events. .PI is the domain of P[]; hence only members of .PI can be placed in the dot position of the notation P[ ]. Since .PI is an algebra, if we assume that A and BE .PI, we know that A, A u B, AB, AB, etc., are also members of .PI, and so it makes sense to talk about P[A], P[A u B], P[AB], P[AB], etc. Properties of P[] For each of the following theorems, assume that nand .PI (an algebra of events) are given and P[] is a probability function having domain .PI. Theorem 15 P[4>] =
PROOF

o.
= 4>,
A2 =

Take

Al

4>, A3 = , ... ; then

Ai]

by axiom (iii)

P[q,]

=P

[,Q

= f~P[Ai] = .t,P[q,],
111/

which can hold only if P[4>] =

o.

Theorem 16 If A., ... , An are mutually exclusive events in .PI, then

P[A 1
U ... U

An] =

L P[Ad.
t::::: 1

24

PROBABILITY

PROOF

Let A II + 1 = , A II + 2 = , ... , ; then

U Ai = U Ai Ed,
i=l

II

<Xl

i=::l

and

IIII

Theorem 17 If A is an event in d, then

P[A]
PROOF

=1-

P[A].

A u A=

Q,

and A
P[Q]

(l

A = ; so
P[A]

= P[A u A] =

+ P[A].
IIII

But P[!l]

1 by axiom (ii); the result follows.

Theorem 18 If A and BEd, thenP[A] = P[AB] + P[AB], and P[A - B]

= P[AB] = P[A]
PROOF

- P[AB]. AB u AB, and AB

(l

AB

= ; so

P[A]

= P[AB] +
Ilfl

P[AB].

Theorem 19 For every two events A and BEd, P[A u B] = P[A]

+ P[B] -

P[AB]. P[A 1
U

More generally, for events

II

Ab

A 2 , ... , All

A2

U . ,

u All] =
- ...

j== 1

L P[A j ] - L L P[AiAj] i<j

An]

+ L L L P[AiAjAk]
i<j<k
PROOF

+ (- 1)"+ 1 P[A 1 A 2 ...

(l

A u B

=A u

AB, and A

AB

= ; so
P[AB].

P[A u B]

= P[A] + P[AB] = P[A] + P[B] -

The more general statement is proved by mathematical induction. Problem 16.)

(See

Ilfl

Theorem 20 If A and BEd and A

PROOF

c:

B, then P[A] < P[B].

= BA u

hence P[B] = P[A] + P[BA] > 0 . '

4>;

BA, and BA = A; so B = A u BA, and A ( l BA = P[BA]. The conclusion follows by noting that

fill

PROBABILITY-AXIOMATIC

2S

Theorem 21

Doole's inequality If Ab A z , ... , A" E,d, then

U ...

P[A 1 u' A2
PROOF

u A,,] < P[Ad

+ P[A z ] + ... + P[A,,].

P[A 1

A 2]

= P[A 1 ] + P[A z ] - P[A 1A z ] < P[Ad + P[A z ]'

The proofis completed using mathematical induction. We conclude this sUbsection with one final definition.

IIII

Definition 16 Probability space A probability space is the triplet (0, d, P[]), where 0 is a sample space, d is a collection (assumed to be an algebra) of events (each a subset of 0), and P[] is a probability function with domain d. IIII Probability space is a single term that gives us an expedient way to assume the existence of all three components in its notation. The three components are related; d is a collection of subsets of 0, and P[] is a function that has d as its domain. The probability space's main use. is in providing a convenient method of stating background assumptions for future definitions, theorems, etc. It also ties together the main definitions that we have covered so far, namely, definitions of sample space, event space, and probability.

3.5

Finite Sample Spaces

In previous subsections we formally defined sample space, event, and probability, culminating in the definition of probability space. We remarked there that these formal definitions did not in themselves enable us to compute the value of the probability-for an event A, which is our goal. We said that we had to appropriately model the experiment. In this section we show how this can be done for finite sample spaces, that is, sample spaces with only a finite number of elements or points in them. In certain kinds of problems, of which games of chance are notable examples, the sample space contains a finite number of points, say N = N(O). [Recall that N(A) is the size of A, that is, the number of sample points in A.] Some of these problems can be modeled by assuming that points in the sample space are equally likely. Such problems are the subject to be discussed next. Finite sample space with equa1ly likely points For certain random experiments there is a finite number of outcomes, say N, and it is often realistic to assume that the probability of each outcome is liN. The classical definition of probability is generally adequate for these problems, but we shall show how

26

PROBABILITY

the axiomatic definition is applicable as well. Let W h W 2 , , wN be the N sample points in a finite space O. Suppose that the set function P[] with domain the collection of all subsets of 0 satisfies the following conditions:

(i) P[{w t }]

= P[{w 2 }] =

...

= P[{WN}]'

(ii) If A is any subset of 0 which contains N(A) sample points [has size N(A)], then P[A] = N(A)IN. Then it is readily checked that the set function P[] satisfies the three axioms and hence is a probability function.

Definition 17 Equally likely probabIlity fUDction The probability function P[ ] satisfying conditions (i) and (ii) above is defined to be an equally
likely probability function. IIII

Given that a random experiment can be realistically modeled by assuming equally likely sample points, the only problem left in determining the value of the probability of event A is to find N(O) = Nand N(A). Strictly speaking this is just a problem of counting-count the number of points in A and the number of points in O.

EXAMPLE 17 Consider the experiment of tossing two dice (or of tossing one die twice). Let 0 = {(ii' i 2): il = 1,2, ... , 6; i2 = 1,2, ... , 6}. Here i1 = number of spots up on the first die, and ;2 = number of spots up on the second die. There are 6'6 = 36 sample points. It seems reasonable to attach the probability of l6 to each sample point. 0 can be displayed as a lattice as in Fig. 2. Let "A 7 = event that the total is 7; then A7 = {(I, 6), (2, 5), (3,4), (4, 3), (5, 2), (6, I)}; so N(A7) = 6, and P[A 7] = N(A 7 )/N(O) = 36(j = 7;. Similarly P[Aj] can be calculated for Aj = total ofj;j = 2, ... , 12. In this example the number of points in any event A can be easily counted, and so P[A] can be evaluated for any event A. IIII If N(A) and N(O) are large for a given random experiment with a finite number of equally likely outcomes, the counting itself can become a difficult problem. Such counting can often be facilitated by use of certain combinatorial formulas, some of which will be developed now. Assume now that the experiment is of such a nature that each outcome can be represented by an n-tuple. The above example is such an experiment; each outcome was represented by a 2-tuple. As another example, if the experiment is one of drawing a sample of size n, then n-tuples are particularly

PROBABIUTY-AXIOMATIC

27

5 4
3

FIGURE 2

:1~: : : _.__________~
1

;1

useful in recording the results. The terminology that is often used to describe a basic random experiment known generally by sampling is that of balls and urns. It is assumed that we have an urn containing, say, M balls, which are numbered 1 to M. The experiment is to select or draw balls from the urn one at a time until n balls have been drawn. We say we have drawn a sample of size n. The drawing is done in such a way that at the time of a particular draw each of the balls in the urn at that time has an equal chance of selection. We say that a ball has been selected at random. Two basic ways of drawing a sample ~are with replacement and without replacement, meaning just what the words say. A sample is said to be drawn with replacement, if after each draw the ball drawn is itself returned to the urn, and the sample is said to be drawn without replacement if the ball drawn is not returned to the urn. Of course, in sampling without replacement the size of the sample n must be less than or equal to M, the original number of balls in the urn, whereas in sampling with replacement the size of sample may be any positive integer. In reportjng the results of drawing a sample of size n, an n-tuple can be used; denote the n-tuple by (Zh . , zn), where Zi represents the number of the ball drawn on the ith draw. In general, we are interested in the size of an event that is composed of points that are n-tuples satisfying certain conditions. The size of such a set can be compute.d as follows; First determine the number of objects, say N I , that may be used as the first component. Next determine the number of objects, say N 2 , that may be used as the second component of an n-tuple given that the first component is known. (We are assuming that N2 does not depend on which object has occurred as the first component.) And then determine the number of objects, say N 3 , that may be used as the third component given that the first and second components are known. (Again we are assuming N3 does not

28

PROBABILITY

depend on which objects have occurred as the first and second components.) Continue in this manner until N n is determined. The size N(A) of the set A of n-tuples then equals Nl N2 ... N n

EXAMPLE 18 The total number of different ordered samples of n balls that can be obtained by drawing balls from an urn containing M distinguishable balls (distinguished by numbers 1 to M) is M n if the sampling is done with replacement and is M(M - 1) ... (M - n + 1) if the sampling is done without replacement. An ordered sample can be represented by an n-tuple, say (Zl,"" zn), where zJis the number of the ball obtained on the jth draw and the total number of different ordered samples is the same as the total number of n-tuples. In sampling with replacement, there are M choices of numbers for the first component, M choices of numbers for the second component, and finally M choices for the nth component. Thus there are M n such n-tuples. In sampling without replacement, there are M choices of numbers for the first component, M - 1 choices for the second, M - 2 choices for the third, and finally M - n + 1 choices for the nth component. In total, then, there are M(M - 1)(M - 2) ... (M - n + 1) such n-tuples. M(M - 1) ... (M - n + 1) is abbreviated (M)n (see Appendix A). II11

EXAMPLE 19 Let S be any set containing M elements. How many subsets does Shave? First let us determine the number of subsets of size n that S has. Let Xn denote this number, that is, the number of subsets of S of size n. A subset of size n is a collection of n objects, the objects not arranged in any particular order. For example the subset {Sl1 ss, J7} is the same as the subset {s 5, S1, S7} since they contain the same three objects. If we take a given subset of S which contains n elements, n! different ordered samples can be obtained by sampling from the given subset without replacement. If for each of the Xn different subsets there are n! different ordered samples of size n, then there are (n !)xn different ordered samples of size n in sampling without replacement from the set S of M elements. But we know from the previous example that this number is (M)n; hence (n!)xn = (M)n, or
Xn

(M)n

n!

= (M) = number of subsets of size n that may be formed

n
from the elements of a set of size M. (1)

PROBABlUTY-AXIOMATIC

29

~ n . The total number of subsets of S, where S is a set of SIze M, IS i..J "=0 This includes the empty set (set with no elements in it) and the whole set, both of which are subsets. Using the binomial theorem (see Appendix A)
with we see that
2M

(M)

= b = 1,

= JJ~);

(2)

thus a set of size M has 2M subsets.

11/1

EXAMPLE 20 Suppose an urn contains M balls numbered 1 to M, where the first K balls are defective and the remaining M - K are nondefective. The experiment is to draw n balls from the urn. Define AA; to be the event that the sample of n balls contains exactly k defectives. There are two ways to draw the sample: (i) with replacement and (ii) without replacement. We are interested in P[A k ] under each method of sampling. Let our sample space Q = {(z., . .. , zJ: zJ = number of the ball drawn on the jth draw}. Now
P[A] _ N(AJ k N(Q)'

From Example 18 above, we know N(Q) = M"under (i) and N(Q) = (M)n under {ii). AA; is that subset of Q for which exactly k of the z/s are ball numbers 1 to K inclusive. These k ball numbers must fall in some subset of k positions from the total number of n available positions. There are (:) ways of selecting the k positions for the ball numbers 1 to K inclusive to fall in. For each of the

(~)

different positions, there are

A;

K.(M - K)-k

different n-tuples for case (i) and (K)k(M - K)n case (ii). Thus A. has size

different n-tuples for case (i) and size

(Z)Kk(M - K)" for

(Z)(Kl.{M - K).-. for case (ii); so,

30

PROBABIUTY

in sampling with replacement, and

(~)(K).(M - K)._.
P[A k ]

= . .

) (M n

(4)

in sampling without replacement. as

This latter formula can be rewritten

(!)(~=:)
P[A.l =
(

'!:) .

(5)

It might be instructive to derive Eq. (5) in another way. Suppose that our sample space, denoted now by 0', is made up of subsets of size n, rather than n-tuples; that is, 0' = {{Zb ... , zn}: Zl' .. , Zn are the numbers

on the

balls drawn}. There are

=

('!:) subsets of size

of the M balls;

so N(fi')

(~)

If it is assumed that each of these subsets of size n is

just as likely as any other subset of size n (one can think of selecting all n balls at once rather than one at a time), then P[Ak] = N(A~)/N(O'). Now N(A~) is the size of the event consisting of those subsets of sizenwhichcontain exactly k balls from the balls that are numbered I to K inclusive. The k balls from the balls that are numbered I to K can be selected in

(!) ways, and the remaining

K

n - k balls from the balls that are numbered

to M

(!)(~ :=

n,

can be selected in

n_ k (M-K)

ways; hence N(AiJ =

and finally

P[A.]

N(AJ/N(fi')

(!) (~:= (:J

n.

We have derived the probability of exactly k defectives in sampling without replacement by considering two different sample spaces; one sample space consisted of n-tuples, the other consisted of subsets of size

n.
K

To aid in remembering the formula given in Eq. (5), note that M - K = M and k + n - k = n; i.e., the sum of the "upper" terms

PROBABlUfV-AXIOMATlC

31

in the numerator equals the" upper" term in the denominator, and the sum of the" lower" terms in the numerator equals the" lower" term in the denominator. // / /

EXAMPLE 21 The formula given in Eq. (5) is particularly useful to calculate certain probabilities having to do with card games. For example, we might ask the probability that a certain 13-card hand contains exactly 6 spades. There are M = 52 total cards, and one can model the card shuffling and dealing process by assuming that the 13-card hand represents a sample of size 13 drawn without replacement from the 52 cards. Let A6 denote the event of exactly 6 spades. There are a total of 13 spades (defective balls in sampling terminology); so 13) (52 - 13) (6 13 - 6
PIA
6 ]

(ii)

by Eq. (5).

IIII

Many other formulas for probabilities of specified events defined on finite sample spaces with equally likely sample points can be derived using methods of combinatorial analysis, but we will not undertake such derivations here. The interested reader is referred to Refs. 10 and 8. Finite sample space without equally likely points We saw for finite sample spaces with equally likely sample points that P[A] = N(A)IN(o.) for any event A. For finite sample spaces without equally likely sample points, things are not quite as simple, but we can completely define the values of P[A] for each of the 2N (Q) events A by specifying the value of P[ ] for each of the N = N(o.) elementary events. Let 0. = {WI' ... , WN}, and assume Pj =P[{Wj}] for j = 1, ... , N. Since
I =

p[n] pL0/en j}]

=
N
j="'l

it/[{en)].
1.

L Pj =

For any event A, define P[A] = J:.Pj' where the summation is over those Wj belonging to A. It can be shown that P[ ] so defined satisfies the three axioms and hence is a probability function.

32

PROBABILITY

EXAMPLE 22 Consider an experiment that has N outcomes, say (01; (02' , (ON' where it is known that outcome (OJ+l is twice as likely as outcome (OJ' wherej = 1, ... , N - 1; that is, Pj+ 1 = 2Pi' where Pi P[{(Ol)]' Find prAll, where A.k = {(Oh (02' , (Ok}' Since

j=1

L Pj = j:::::l L 211 Pi = 2N -1

PI

= Pl(1 + 2 + 22 + ... + 2N -

= Pl(2 N

1)

= 1,

and hence

C>

3.6

Conditional Probability and Independence

In the application of probability theory to practical problems it is not infrequent that the experimenter is confronted with the following situation: Such and such has happened; now what is the probability that something else will happen? For example, in an experiment of recording the life of a light bulb, one might be interested in the probability that the bulb will last 100 hours given that it has already lasted for 24 hours. Or in an experiment of sampling from a box containing 100 resistors of which 5 are defective, what is the probability that the third draw results in a defective given that the first two draws resulted in defectives? Probability questions of this sort are considered in the framework of conditional probability, the subject that we study next. Conditional probability We begin by assuming that we have a probability space, say (n, d, P[ ]); that is, we have at hand some random experiment for which a sample space n, collection of events d, and probability function P[ . ] have all been defined. Given two events A and B, we want to define the conditional probability of even"t A given that event B has occurred. Definition 18 Conditional probability Let A and B be two events in d of the given probability space (n, d, P[ ]). The conditional probability of event A given event B, denoted by P[A IB], is defined by
P[A IB]

= ~AB]
P[B]

if

P[B] > 0,

(6)

and is left undefined if P[B] = O.

1111

PROBABILITY-AXIOMATIC

33

Remark A formula. that is evident from the definition is P[AB] = P[A IB]P[B] = P[BI A]P[A] if both P[A] and P[B] are nonzero. This formula relates P[A IB] to P[BI A] in terms of the unconditional probabilities P[A] and P[B]. IIII
We might note that the above defmition is compatible with the frequency approach to probability, for if one observes a large number, say N, of occurrences of a random experiment for which events A and B are defined, then P[A IB] represents the proportion of occurrences in which B occurred that A also occurred, that is,
P[A IB] = NAB, NB

where N B denotes the number of occurrences of the eve~t B in the N occurrences of the random experiment and NAB denotes the number of occurrences of the event A n B in the N occurrences. Now P[AB] = NABIN, and P[B] = NBIN; so
P[AB] P[B]

NABIN = NAB NBIN NB

= P[AIB]
'

consistent with our definition.

EXAMPLE 23 Let 0 be any finite sample space, d the collection of all subsets of 0, and P[] the equally likely probability function. Write N = N(O). For events A and B,
P[AIB] = P[AB] = N(AB)IN P[B] N(B)! N '

where, as usual, N(B) is the size of set B. So for any finite sample space with equally likely sample points, the values of P[A IB] are defined for any two events A and B provided P[B] > O. IIII

EXAMPLE 24 Consider the experiment of tossing two coins. Let 0 = {(H, H), (H', T), (T, H), (T, T)}, and assume that each point is equally likely. Find (i) the probability of two heads given a head on the first coin and (ii) the probabiUty of two heads given at least one head. Let Al = {head on first coin} and A2 = {head on second coin}; then the probability of two heads given a head on the first coin is
P[A A
1 2

IA

] = P[AIA2Ad = P[A 1 A 2] t 1 P[A d P[Ad!

= =!

2.

34

PROBABILITY

The probability of two heads given at least one head is

P [ AiA2 1 Ai
U

A2 ] =

P[AIA2 (\ (A~l U 04 2)] P[A 1 A 2] ! = P[A 1 U A 2] P[A 1 u A 2] i

1 3

We obtained numerical answers to these two questions, but to do so we had to model the experiment; we assumed that the four sample points were equally likely. When speaking of conditional probabilities we are conditioning on some given event B; that is, we are assuming that the experiment has resulted in some outcome in B. B, in effect, then becomes our" new" sample space. One question that might be raised is: For given event B for which P[B] > 0, is P['I B] a probability function having d as its domain? In other words, does P[ IB] satisfy the three axioms? Note that:

(i) P[A IB]

(ii)

P[AB]IP[B] > 0 for every A Ed.

P[OI B] = P[OB]IP[B] = P[B]IP[B] = 1.

(iii)
00

If AI' A 2 ,

is a sequence of mutually exclusive events in .91 and

i=:l

U Ai Ed, then

Hence, P['I B] for given B satisfying P[B] > 0 is a probability function, which justifies our calling it a conditional probability. P[ IB] also enjoys the same properties as the unconditional probability. The theorems listed below are patterned after those in Subsec. 3.4. Properties of Pl IB] Assume that the probability space (n, .91, P[]) is given, and let BEd satisfy P[B] > O. Theorem 22 P[q,IB] Theorem 23

O.

IIII

If At, ... , An are mutually exclusive events in .91, then

PlAt
U ... U

Ani B] = i;;;;;; I

P[Ad B].
1

IIII

Theorem 24

If A is an event in d, then
P[AI B]

= 1-

P[A I B].

IIII

PROBABIUTY-AXlOMATIC

3S

Theorem 25 If At and A2 E d, then

P[Ad B] = P[A1A2IB]

+ P[A t A21B].

////

Theorem 26 For every two events At and A2 E d,

PlAt u A2IB]

= P[A1IB] +P[A 2IB] C

p[A1A2IB].

/1//

Theorem 27 If A1 and A2 E d and Al

A2 , then

P[AII B] < P[A21 B].

1//1

Theorem 28 If A 1, A 2 , ... , An Ed, then

II

P[A I

A2

U U

AnI B ] < LP[AIIB].

)=1

1/11

Proofs of the above theorems follow from known properties of P[] and are left as exercises. There are a number of other useful formulas involving conditional probabilities that we will state as theorems. These will be followed by examples.

Theorem 29 Theorem of total probabilities For a given probability space (n, d, P[ ]), if B1, B 2 , , Bn is a collection of mutually disjoint
events in d satisfying

n=
n
j=1

UB
)=1

and P[Bj ] > O for j

1, "" n, then

for- every A E d, P[A] = L P[A IBj]P[Bj ].

PROOF

Note that A =

j= 1

U ABj

and the AB/s are mutually disjoint;

hence

1//1
Corollary For a given probability space (n, d, P[]) let BEd satisfy 0 < P[B] < 1; then for every A Ed
P[A] = P[AIB]P[B]

+ P[AIB]P[B].

//1/
1///

Remark Theorem 29 remains true if n =

00.

36

PROBABILITY

Theorem 29 (and its corollary) is particularly useful for those experiments that have stages; that is, the experiment consists of performing first one thing (first stage) and then another (second stage). Example 25 provides an example of such an experiment; there, one first selects an urn and then selects a ball from the selected urn, For such experiments, if Bj is an event defined only in terms of the first stage and A is an event defined in terms of the second stage, then it may be easy to find P[Bj ]; also, jt may be easy to find peA IBj ], and then Theorem 29 evaluates peA] in terms of P[Bj ] and peA IB j ] for j = 1, ... , n, In an experiment consisting of stages it is natural to condition on results of a first stage. Theorem 30 Bayes' formula For a given probability space (0, .91, Pl D, if B 1 , B 2 , , Bn is a collection of mutually disjoint events in .91 satisfying

n = U Bj
j:::l

and P[Bj ] > 0 for j = 1, ... , n, then for every A Ed for which

peA] >0 P[B"IA]

nP[AIBk]P[Bk ]
j= 1

j]

2: P[AIBj]P[B

PROOF
,

..

by using both the definition of conditional probability and the theorem of total pro babi Ii ties. IIII Corollary For a given probability space (0, d, P[]) let A and BE d satisfy peA] > 0 and 0 < PCB] < I ; then
P B/ _ P[A/B]P[B] [ A] - P[A' B}P[B] + peA IB]P[B]'

1I11 11II

Remark Theorem 30 remains true if n =

00.

As was the case with the theorem of total probabilities, Bayes' formula is also particularly useful for those experiments consisting of stages. If Bj , j = 1, ... , n, is an event defined in terms of a first stage and A is an event defined in terms of the whole experiment including a second stage, then asking for P[BkIA] is in a sense backward; one is asking for the probability of an event

PROBABILITY-AXJOMATIC

37

defined in terms of a first stage of the experiment conditioned on what happens in a later stage of the experiment. The natural conditioning would be to condition on what happens in the first stage of the experiment, and this is precisely what Bayes' formula does; it expresses P[Bkl A] in terms of the natural conditioning given by P[AIB j ] and P[Bj],j= 1, ... , n. Multiplication rule For a given probability space (0, d, P[ D, let AI,"" An be events belonging to d for which P[A I , An- tl > 0; then Theorem 31
P[A 1 A 2

An]

= P[AdP[A 2

AtlP[A 3 1 A 1 A 2 ]

PlAn IAl ... An-d

The proof can be attained by employing mathematical induction and is left as an exercise. If/I
PROOF

As with the two previous theorems, the multiplication rule is primarily useful for experiments defined in terms of stages. Suppose the experiment has n stages and A J is an event defined in terms of stage j of the experiment; then .f~f[Ajl AIA2 .. , Aj-d is the conditional probability of an event described in . terms of what happens on stage j conditioned on what happens on stages 1, 2, ... , j - 1. The multiplication rule gives PlAt A 1 .. An] in terms of the natural conditional probabilities P[A j IA 1A 2 '" Aj-tl forj= 2, ... , n.

EXAMPLE 25 There are five urns, and they are numbered I to 5. Each urn contains 10 balls. Urn i has i defective balls and 10 - i nondefective balls, i = 1, 2, ... , 5. For instance, urn 3 has three defective balls and seven nondefective balls. Consider the following random experiment: First an urn is selected at random, and then it ball is selected at random from the selected urn. (The experimenter does not know which urn was selected.) Let us ask two questions: (i) What is the probability that a defective ball will be selected? (ii) If we have already selected the ball and noted that it is defective, what is the probability that it came from urn 5? Let A denote the event that a defective ball is selected and B t the event that urn i is selected, i = I, ... , 5. Note that P[B,l = ~, i 1, ... ,5, andP[AIBi ] = i/lO, i= 1, ... , 5. Question (i) asks, What is P[A]? Using the theorem of total probabilities, we have
SOLUTION

peA]

= i~lP[ArB,]p[Ba = l~ 10'"5 = 50 t~l'

ill

5.

1 56 3 = 50"2 = 10'

38

PROBABILITY

Note that there is a total of 50 balls of which 15 are defective! Question (ii) asks, What is P[Bsl A]? Since urn 5 has more defective balls than any of the other urns and we selected a defective ball, we suspect that P[Bsl A] > P[B i IA] for i = 1, 2, 3, or 4. In fact, we suspect P[Bsl A] > P[B41 A] > ... > P[BII A]. Employing Bayes' formula, we find
P[BsIA]

;[AIBs]P[Bs]

=t~!=~.
To

I
i= I

P[A IBtJP[BtJ
.t

Similarly,
P[B IA]
k

= (kilO)
10

= !5...

15'

= 1, ... , 5,

substantiating our suspicion. Note that unconditionally all the B/s were equally likely whereas, conditionally (conditioned on occurrence of event A), they were not. Also, note that
s
k=l

IP[BkIA]

= k=115 I -=Ik=--=l. 15k=l 152

skI

1 56

IIII

EXAMPLE 26 Assume that a student is taking a multiple-choice test. On a given question, the student either knows the answer, in which case he answers it correctly, or he does not know the answer, in which case he guesses hoping to guess the right answer. Assume that there are five multiple-choice alternatives, as is often the case. The instructor is confronted with this problem: Having observed that the student got the correct answer, he ,wishes to know what is the probability that the student knew the answer. Let p be the probability that the stydent will know the answer and 1 - P the probability that the student guesses. Let us assume that the probability that the student gets the right answer given that he (This may not be a realistic assumption since even though the guesses is student does not know the right answer, he often would know that certain alternatives are wrong, in which case his probability of guessing correctly should be better than ~.) Let A denote the event that the student got the right answer and B denote the event that the student knew the right answer. We are seeking P[BI A]. Using Bayes' formula, we have

t.

P[BI A] Note that

P[A IB]P[B] P[A IB]P[B] + P[A IB]P[B]

1 .p

= 1 . P + t(1 - p)"

p+t~I-P)";::.P.

IIII

PROBABlUTY-AXIOMATIC

39

EXAMPLE 27 An urn contains ten balls of which three are black and seven are white. The following game is played: At each trial a ball is selected at random, its color is noted, and it is replaced along with two additional balls of the same color. What is the probability that a black ball is selected in each of the first three trials? Let Bi denote the event that a black ball is selected on the ith trial. We are seeking P[B1 B 2 B3]' By the mul tiplication r"ule,
P[B 1 B 2 B 3]
\
'

= P[B 1 JP[B 2 I'BdP [B 3 IB 1B 2 ] = 130

/2 174

/6'

IIII

EXAMPLE 28 Suppose an urn contains M balls of which K are black and M - K are white. A sample of size n is drawn. Find the probability that the jth ball drawn is black given that the sample contains k black balls. (We intuitively expect the answer to be kin.) We have to consider sampling (i) with replacement and (ii) without replacement.
SOLUTION

Let Ak denote the event that the sample contains exactly k black balls and Bj denote the event that the jth ball drawn is black. We seek P[Bjl Ak]' Consider (i) first.

P[AkJ =

n) Kk(M - K)"-k (k M(I

and

(n 1) K kP[A kIBJ.J =
k- 1

(M _ K)"-k 1 M"-

by Eq. (3) of Subsec. 3.5. any j. Hence,

Since the balls are replaced, P[Bj ] = KIM for

For case (ii),

P[ A.J =

(~)(~ ~ ~)
( '::)

and

K- 1)(M - K) ( P[A IB.J = k - 1 n - k (M _ 1)

k
J

n- 1

j-1

by Eq. (5) of Subsec. 3.5.

P[Bj

i == 0

L P[Bjl CdP[Ci ] ,

where C i denotes Note that

the event of exactly i black ba~ls in the firstj - 1 draws.

40

PROBABILITY

and
P[B., C.]
J
t

=-M-j+l'

K-i

and so

Finally,
PCB _I A]
J

, 'P[A

kI

[(K l)(M K)j(M 1)] K Bj]P[BJ = k- 1 n- k n- 1 Ai = ~.

(~)('~

P[Akl

= f) j(~)

n
.

Thus we obtain the same answer under either method of sampling. "t-'/IIF, "

Independence of events If P[A IB] does not depend on evem: B, that is, P[A, B] = P[A], then it would seem natural to say that event A is independent of event B. This is given in the following definition. Definition 19 Independent events For a given probability space (0, &/, P[]), let A and B be two events in.!il. Events A and Bare defined to be independent if and only if anyone of the following conditions is satisfied:
(i)

P[AB] = P[A]P[B].

(ii) P[A, B] = P[A] if P[B] > O. (iii) P[B IA]

P[B] if PtA] > O.

IIII

Remark Some authors use" statistically independent," or "stochasti/111 cally independent," instead of" independent."
To argue the equivalence of the above three conditions, it suffices to show that (i) implies (il), Oi) implies (iii), and (iii) implies (i). If P[AB] P[A]P[B], then P[A, B] = P[AB]IP[B] = P[A]P[B]fP[B] = P[A] for P[B] > 0; so (i) implies (ii). If PtA IB] = P[A], then P[BI A] = P[A, B]P[B]IP[A] = P[A]P[B]IP[A] = P[B] for P[A] > 0 and P[B] > 0; so (ii) implies (iii). And if P[BI A] = P[B], then P[AB] = P[B,A]P[A] = P[B]P[A] for P[A] > O. Clearly P[AB] = P[A]P[B] if P[A] = 0 or P[B] = o.

PROBABILITY-AXIOMATIC

41

EXAMPLE 29 Consider the experiment of tossing two dice. Let A denote the event of an odd total, B the event of an ace on the first die, and C the event of a total of seven. We pose three problems: (i) Are A and B independent? Oi) Are A and C independent? (iii) Are Band C independent? We obtain P[A IB] = 1 = P[A], P[A I C] = I :# P[A] = 1, and P[CI B] = 1; = P[C] = !; so A and B are independent, A is not independent of C, and Band C are independent. IIII The property of independence of two events A and B and the property that A and B are mutually exclusive are distinct, though related, properties. For example, two mutually exclusive events A and B are independent if and only if P[A]P[B] = 0, which is true if and only if either A or B has zero probability. Or if P[A] :# 0 and P[B] =F 0, then A and B independent implies that they are not mutually exclusive, and A and B mutually exclusive implies that they are not independent. Independence of A and B implies independence of other events as wel1.

Theorem 32 If A and B are two independent events defined on a given

probability space (0, d, P[]), then A and 13 are independent, A and B are independent, and .If and 13 are independent.
PROOF

P[ABJ = P[A] - P[AB]

= P[A]

- P[A]P[B]

= P[A](l

- P[BD

=
P[A]P[B].

Similarly for the others.

IIII

The notion of independent events may be extended to more than two events.

Definition 20 Independence of several events For a given probability

space (0, d, Pl']), let A., A 2 , , An be n events in d. A 2 , , All are defined to be independent if and only if
P[A,A j ]

Events Ab

= P[Ai]P[A~]

for i :# j for i:# j,j:# k, i =F k

P[Ai1 j A d = P[A f ]P[A j ]P[A k l

P[.OI AI] = tIl PtA,].

. .

1111

42

PROBABILITY

One might inquire whether all the above conditions are required in the definition. For instance, does P[A I A 2 A 3] = P[AtlP[A 2 ]P[A 3 ] imply P[A I A 2 ] = P[AtlP[A 2 ]? Obviously not, since P[A 1A 2 A 3 ] = P[AdP[A 2 ]P[A 3 ] if P[A 3] = 0, but P[A I A 2 ] #: P[AdP[A 2 ] if At and A2 are not independent. Or does pairwise independence imply independence? Again the answer is negative, as the following example shows.

EXAMPLE 30 Pairwise independence does not imply independence. Let Al denote the event of an odd face on the first die, A2 the event of an odd face on the second die, and A3 the event of an odd total in the random experiment that consists of tossing two dice. P[AdP[A 2 ] = ! . ! = P[A 1 A 2], P[AtlP[A 3] = 1 '1 = P[A31 AtlP[Ad = P[A I A 3], and P[A 2 A 3 ] = i = P[A 2 ]P[A 3 ]; so Ab A 2 , and A3 are pairwise independent, However P[A 1 A 2 A 3 ] = 0 #: ! = P[AdP[A 2 ]P[A 3 ]; so Ah A 2 , and A3 are not independent. IIII In one sense, independence and conditional probability are each used to find the same thing, namely, P[AB], for P[AB] = P[A]P[B] under independence and P[AB] = P[A IB]P[B] under nonindependence. The nature of the events A and B may make calculations of P[A], P[B], and possibly P[A IB] easy, but direct calculation of P[AB] difficult, in which case our formulas for independence or conditional probability would allow us to avoid the difficult direct calculation of P[AB]. We might note that P[AB] = P[A IB]P[B] is valid whether or not A is independent of B provided that P[A IB] is defined. The definition of independence is used not only to check if two given events are independent but also to model experiments. For instance, for a given experiment the nature of the events A and B might be such that we are willing to assume that A and B are independent; then the definition of independence gives the probability of the event A n B in terms of P[A] and P[B]. Similarly for more than two events.

EXAMPLE 31 Consider the experiment of sampling with replacement from an urn containi ng M balls of which K are black and M K white. Since balls are being replaced after each draw, it seems reasonable to assume that the outcome of the second draw is independent of the outcome of the first. Then P[two blacks in first two draws] = P[black on first draw]P[black on second draw] = (KIM)2, IIII

PROBLEMS

43

PROBLEMS
To solve some of these problems it may be necessary to make certain assumptions, such as sample points are equally likely, or trials are independent, etc., when such assumptions are not explicitly stated. Some of the more difficult problems, or those that require'special knowledge, are marked with an *. lOne urn contains one black ball and one gold ball. A second urn contains one white and one gold ball. One ball is selected at random from each urn. (a) Exhibit a sample space for this experiment. (b) Exhibit the event space. (c) What is the probability that both balls will be of the same color? (d) What is the probability that one ball will be green? 2 One urn contains three red balls, two white balls, and one blue ball. A second urn contains one red ball, two white balls, and three blue balls. (a) One ball is selected at random from each urn. (i) Describe a sample space for this experiment. (ii) Find the probability that both balls will be of the same color. (iii) Is the probability that both balls will be red greater than the probability that both will be white? (b) The balls in the two urns are mixed together in a single urn, and then a sample of three is drawn. Find the probability that all three colors are represented, when (i) sampling with replacement and (ii) without replacement. 3 If A and B are disjoint events, P[A] =.5, and P[A u B] = .6, what is P[B]? 4 An urn contains five balls numbered 1 to 5 of which the first three are black and the last two are gold. A sample of size 2 is drawn with replacement: Let Bl denote the event that the first ball drawn is black and B2 denote the event that the second ball drawn is black. (a) Describe a sample space for the experiment, and exhibit the events B 1 , B 2 , and B 1 B 2 (b) Find P[B1], P[B 2 ], and P[B1B2]' (c) Repeat parts (a) and (b) for sampling without replacement. 5 A car wit~ six spark plugs is known to have two malfunctioning spark plugs. If two plugs are pulled at random, what is the probability of getting both of the malfunctioning plugs ? 6 In an assembly-line operation, 1 of the items being produced are defective. If three items are picked at random and tested, what is the probability: (a) That exactly one of them will be defective? (b) That at least one of them will be defective? 7 In a certain game a participant is allowed three attempts at scoring a hit. In the three attempts he must alternate which hand is used; thus he has two possible strategies: right hand, left hand, right hand; or left hand, right hand, left hand. His chance of scoring a hit with his right hand is .8, while it is only .5 with his left hand. If he is successful at the game provided that he scores at least two hits in a row, what strategy gives the better chance of success? Answer the same

44

PROBABILITY

9
10

11 12 13 14 15 16 17

18

19

20

question if .8 is replaced by PI and .5 by P2. Does your answer depend on PI and P2? . (a) Suppose that A and B are two equally strong teams. Is it more probabfe seven? ~ that A will beat B in three games out of four or in five games out of (b) Suppose now that the probability that A beats B in an individual game is p. Answer part (a). Does your answer depend on p? If P[A] = t and P[B] = !, can A and B be disjoint? Explain. Prove or disprove; If P[A] =P[B] =p, thenP[AB] <p2. Prove or disprove: If P[A] = P[B] then A = B. Prove or disprove: If P[A] = 0, then A = rp. Prove or disprove: If P[A] = 0, then P[AB] =0. Prove: If P[A] = IX and P[B] = fl, then P[AB] > 1 - IX -fl. Prove properties (i) to (iv) of indicator functions. Prove the more general statement in Theorem 19. Exhibit (if such exists) a probability space, denoted by (0, d, P[ D, which satisfies. the following. For Al and A2 members ofd, if P[Atl = P[A 2], then Al = A 2 . Four drinkers (say I, II, III, and IV) are to rank three different brands of b&r (say A, B, and C) in a blindfold test. Each drinker ranks the three beers as.-l. (for the peer he likes best), 2, and 3, and then the assigned ranks of each brand of beer,are summed. Assume that the drinkers really cannot discriminate between beers so that each is assigning his rankings at random. (a) What is the probability that beer A will receive a total score of 4? (b) What is the probability that some beer will receive a total score of ~? (c) What is the probability that some beer will receive a total score of 5 or less? The following are three of the classical problems in probability. (a) Compare the probability of a total of 9 with a total of 10 when trrreefair dice are tossed once (Galileo and Duke of Tuscany). (b) Compare the probability of at least one 6 in 4 tosses of a fair die witli . .,. the, probability of at least one double-6 in 24 tosses of two fair dice (Chevalier de Mere). (c) Compare the probabiJity of at least one 6 when six dice are rolled witi- the probability of at least two 6s when twelve dice are rolled (Pepys to Newton). A seller has a dozen small electric motors, two of which are faulty .. Acust~mff is interested in the dozen motors. The seller can crate the motors with all twelve in ,/L one box or with six in each of two boxes; he knows that the customer will inspect twO of the twelve motors if they are all crated in one box and one motor from each' of the two smaller boxes if they are crated six each to two smaller boxes. He has three strategies in his attempt to sell the faulty motors: (i) crate all twelve in one box; (ii) put one faulty motor in each of the two smaller box~; or (iii) put both of the faulty motors in one of the smaller boxes and no faulty motor~ in the other. What is the probability that the customer will not inspect a faulty motor under each of the three strategies?
,

".

PROBLEMS

45

A sample of five objects is drawn from a larger population of N objects (N 5). Let Nw or N wo denote the number of different samples that could be drawn depending, respectively, on whether sampling is done with or without replacement. Give the values for N w and Nwo Show that when N is very large, these two values are approximately equal in the sense that their ratio is close to 1 but not in the sense that their difference is close to O. 22 Out of a .!oup of 25 persons, what is the probability that all 25 will have different birthdays? (Assume a 365-day year and that all days are equally likely.) 23 A bridge player knows that his two opponents have exactly five hearts between the two of them. Each opponent has thirteen cards. What is the probability that there is a three-two split on the hearts (that is, one player has three hearts and the other two)? 24 (a) If r balls are randomly placed into n urns (each ball having probability lin of going into the first urn), what is the probability that the first urn will contain exactly k balls? " (b) Let n -'1-Cl) and r -7 Xl while r/n = m remains constant. Show that the ~ probapility you calculated approaches e .. m k /k!. ,}S' )(:{biased coin has probability p of landing heads. Ace, Bones, and Clod toss the coin successively, Ace tossing first, until a head occurs. The person who tosses the first head wins. Find the probability of winning for each. *26 It is told that in certain rural areas of Russia marital fortunes were once told in the following way: A girl would hold six strings in her hand with the ends protruding abQve and below; a friend would tie together the six upper ends in pairs and then tie together the six lower ends in pairs. If it turned out that the friend had tied ~he six strings into at least one ring, this was supposed to indicate that the girl ''Would get married within a year. What is the probability that a single ring will ,be formed when the strings are tied at random? What is the probability that at 'least one ring will be formed? Generalize the problem to 2n strings. 27 Mr. Bandit, a well-known rancher and not so well-known part-time cattle rustler, lls twenty head of cattle ready for market. Sixteen of these cattle are his own tfiilnd consequently bear his own brand. The other four bear foreign brands. Mr. Bandit knows that the brand inspector at the market place checks the brands of .20 percent of the cattle in any shipment. He has two trucks, one which will haul all twenty <!attle at once and the other that will haul ten at a time. Mr. Bandit' feels that he has four djfferent strategies to follow in his attempt to market the cattle without getting caught. The first is to sell all twenty head at once; the others are to sell ten head on two different occasions, putting all four stolen cattle in one set of ten, or three head in one shipment and one in the other, or two head in each of the shipments of ten. Which strategy will minimize Mr. Bandit's proba~ility 6i" getting caught, and what is his probability of getting caught under each strategy? 28 Show that the formula of Eq. (4) is the same as the formula of Eq. (5).
21
",-W.

lll

46

PROBABIliTY

29 Prove Theorem 31. 30 Either prove or disprove each of the following (you may assume that none of the events has zero probability): (a) If P[A IB] > P[A], then P[BI A] > P[B]. (b) If P[A] > P[B], then P[A IC] > p[BI C]. 31 A certain computer program will operate using either of two subroutines, say A and B, depending on the problem; experience has shown that subroutine A will be used 40 percent of the time and B will be used 60 percent of the time. If A is used, then there is a 75 percent probability that the program will run before its time limit js exceeded; and jf B is used, there is a 50 percent chance that it will do so. What is the probability that the program will run without exceeding the time limit? 32 Suppose that it is known that a fraction .001 of the people in a town have tuberculosis (TB). A tuberculosis test is given with the following properties: If the person does have TB, the test will indicate it with a probability .999. If he does not have TB, then there is a probability .002 that the test will erroneously indicate that he does. For one randomly selected person, the test shows that he has TB. What is the probability that he really does? *33 Consider the experiment of tossing two fair regular tetrahedra (a polyhedron with four faces numbered 1 to 4) and noting the numbers on the downturned faces. (a) Give three proper events (an event A is proper if 0 < P[A] < 1) which are independent (if such exist). (b) Gjve three proper events which are pairwise independent but not independent (if such exist). (c) Give four proper events which are independent (if such exist). 34 Prove or disprove: (a) If A and B are independent events, then P[ABI C] = P[A IC]p[BI C]. (b) If p[A IB] =P[B], then A and B are independent. 35 Prove or disprove: (a) If P[A IB] > P[A], then P[BI A] > P[B]. (b) If p[BI A] =P[BI A], then A and B are independent. (c) If a = P[A] and b =P[B], then P[A IB] > (a + b - 1)/b. 36 Consider an urn containing 10 balls of which 5 are black. Choose an integer n at random from the set 1, 2, 3, 4, 5, 6, and then choose a sample of size n without replacement from the urn. Find the probability that all the balls in the sample will be black. 37 A die is thrown as long as necessary for a 6 to turn up. Given that the 6 does not turn up at the first throw, what is the probabjlity that more than four throws will be necessary? 38 Die A has four red and twO blue faces, and die B has two red and four blue faces. The following game is played: First a coin is tossed once. If it falls heads, the game continues by repeatedly throwing die A; if it falls tails, die B is repeatedly tossed.

PROBLEMS

47

(a) (b)

39

*40

4J

*42

4!

Show that the probability of red at any throw is i. If the first two throws of the die resulted in red, what is the probability of red at the third throw? (c) If red turns up at the first n throws, what is the probability that die A is being used? Urn A contains two white and two black bans; urn B contains. three white and two black bans. One ball is transferred from A to B; one ball is then drawn from B and turns out to be white. What is the probability that the transferred ball was white? It is known that each of four people A, B, C, and D tells the truth in a given instance with probability 1-. Suppose that A makes a statement, and then D says that C says that B says that A was telling the truth. What is the probability that A was actually telling the truth? In a T maze, a laboratory animal is given a choice of going to the left and getti ng food or going to the right and receiving a mild electric shock. Assume that before any conditioning (in trial number I) animals are equally likely to go to the left or to the right. After having received food on a particular trial, the probabilities of going to the left and right become .6 and .4, respectively, on the following trial. However, after receiving a shock on a particular trial, the probabilities of going to the left and right on the next trial are .8 and .2, respectively. What is the probability that the animal will turn left on trial number 2? On trial number 3? In a breeding experiment, the male parent is known to have either two dominant genes (symbolized by AA) or one dominant and one recessive (Aa). These two cases are equally likely. The female parent is known to have two recessive genes (aa). Since the offspring gets one gene from each parent, it will be either Aa or aa, and it will be possible to say with certainty which one. (a) If we suppose one offspring is Aa, what is the probability that the male parent is A A ? (b) If we suppose two offspring are both Aa, what is the probability that the male parent is AA ? (c) If one offspring is aa, what is the probabjJity that the male parent is Aa 1 The constitution of two urns is
I

three black two white

II

four black six white

A draw is made by selecting an urn by a process which assigns probability p to the selection of urn I and probability 1 p to the selection of urn II. The selection of a ball from either urn is by a process which assigns equal probability to all balls in the urn. What value of p makes the probability of obtaining a black ball the same as if a single draw were made from an urn with seven black and eight white balls (all balls equally probable of being drawn)?

48

PROBABILITY

44

45

46

47

48

49 50

51 52 53
54

55

*56

57

Given P[A] = .5 and P[A v B) = .6, find P[B] if: (a) A and B are mutually exclusive. (b) A and B are independent. (c) P[A IB] = .4. Three fair dice are thrown once. Given that no two show the same face: (a) What is the probability that the sum of the faces is 7? (b) What is the probability that one is an ace? Given that P[A] > 0 and P[B] > 0, prove or disprove: (a) If P[A] = P[B], then P[A IB] = P[BI A]. (b) If P[A IB) = p[BI A), then P[A] = P[B]. Five percent of the people have high blood pressure. Of the people with high blood pressure, 75 percent drink alcohol; whereas, only 50 percent of the people without high blood pressure drink alcohol. What percent of the drinkers have high blood pressure? A distributor of watermelon seeds determined from extensive tests that 4 percent of a large batch of seeds will not germinate. He sells the seeds in packages of 50 seeds and guarantees at least 90 percent germination. What is the probability that a given package will violate the guarantee? If A and B are independent, P[A] = t, and P[E] = 1, find P[A u B]. Mr. Stoneguy, a wealthy diamond dealer, decides to reward his son by allowing him to select one of two boxes. Each box contains three stones. In one box two of the stones are real diamonds, and the other is a worthless imitation; and in the other box one is a real diamond, and the other two worthless imitations. If the son were to choose randomly between the two boxes, his chance of getting two real diamonds would be!. Mr. Stoneguy, being a sporting type, allows his son to draw one stone from one of the boxes and to examine it to see if it is a real diamond. The son decides to take the box that the stone he tested came from if the tested stone is real and to take the other box otherwise. Now what is the probability that the son will get two real d!amonds? If P[A] =P[B]=P[BI A) = !, are A and B independent? If A and B are independent and P[A] = P[B] = !, what is P[AE v AB]? If P[B) = P[A IB] = P[CI AB] = i, what is P[ABC]? If A and B are independent and P[A) =P[BIA) = i, what is P[A u B]? Suppose Bt, B 2 , and B3 are mutually exclusive. If P[Bil = ! and P[A IB j ] = j/6 for j = 1, 2, 3, what is P[A]? The game of craps is played by letting the thrower toss two dice until he either wins or loses. The thrower wins on the first toss if he gets a total of 7 or I I ; he loses on the first toss if he gets a total of 2, 3, or 12. If he gets any other total on his first toss, that total is called his point. He then tosses the dice repeatedly untH he obtains a total of 7 or his point. He wins if he gets his point and loses if he gets a total of 7. What is the thrower's probability of winning? In a dice game a player casts a pair of dice twice. He wins if the two totals thrown do not differ by more than 2 with the following exceptions: If he gets a

PROBLEMS

49

3 on the first throw, he must produce a 4 on the second throw; if he gets an ) I on the first throw, he must produce a 10 on the second throw. What is his probability of winning? 58 Assume that the conditional probability that a child born to a couple will be ~ is! mEl fEz ~ where El and Ez are certain small constants, m is the nUmber of male children already born to the couple, andfis the number of female children already born to the couple. (a) What is the probability that the third child will be a boy given that the first two are girls? (b) Find the probability that the first three children will be all boys . . (c) Find the probability of at least one boy in the first three children. (Your answers will be expressed in terms of 81 and 82.) *59 A network of switches a, b, c, and d is connected across the power lines A and B as shown in the sketch. Assume that the switches operate electrically and have independent operating mechanisms. All are controlled simultaneously by the same impulses; that is, it is intended that on an impulse all switches shall close &l;imultaneously. But each switch has a probability P of failure (it will not close when it should).

~--------4C~----~----~~-------4

---(a)

-------- Power lines

What is the probability that the circuit from A to B will fail to close? (b) If ~a line is added on at e, as indicated in the sketch, what is the probability that the circuit from A to B will fail to close? (c) 'If a line and switch are added at e, what is the probability that the circuit from A to B will fail to close?
60

Let Bt, B 2 , , Bn be mutually disjoint, and let B =

J=1

U BJ

II

Suppose P[Bj ] > 0

61

and P(A I B J] = P for j = I, ... , n. Show that peA 1 B] = p. In a laboratory experiment, an attempt is made to teach an animal to turn right in a maze. To aid in the teaching, the animal is rewarded if it turns right on a given trial and punished if it turns left. On the first trial the animal is just as likely to turn right as left. If on a particular trial the animal was rewarded, his probability of turning right on the next trial is PI > !, and if on a given trial the animal was punished, his probability of turning right on the next trial is P2 > Pl. (a) What is the probability that the animal will turn right on the third trial? (b) What is the probability that the animal wHi turn right on the third trial, given that he turned right on the first trial?

50

PROBABILITY

*62 You are to play ticktacktoe with an opponent who on his turn makes his mark by selecting a space at random from the unfilled spaces. You get to mark first. Where should you mark to maximize your chance of winning, and what is your probability of winning? (Note that your opponent cannot win, he can only tie.) 63 Urns I and II each contain two white and two black balls. One ball is selected from urn I and transferred to urn II; then one ball is drawn from urn II and turns out to be white. What is the probability that the transferred ball was white? 64 Two regular tetrahedra with faces numbered 1 to 4 are tossed repeatedly until a total of 5 appears on the down faces. What is the probability that more than two tosses are required? 65 Given P[A] = .5 and P[A v B] = .7: (a) Find P[B] if A and B are independent. (b) Find P[B] if A and B are mutually exclusive. (c) Find P[B] if P[A IB] =.5. 66 A single die is tossed; then n coins are tossed, where n is the number shown on the die. What is the probability of exactly two heads? *67 In simple Mendelian inheritance, a physical characteristic of a plant or animal is determined by a single pair of genes. The color of peas is an example. Let y and 9 represent yellow and green; peas will be green if the plant has the color-gene pair(g, g); they will be yellow if the color-gene pair is (y, y) or (y, g). In view of this last combination, yellow is said to be dominant to green. Progeny get one gene from each parent and are equally likely to get either gene from each parent's pair. If (y, y) peas are crossed with (g, g) peas, all the resulting peas will be (y, g) and yellow because of dominance. If (y, g) peas are crossed with (g, g) peas, the probability is .5 that the resulting peas will be yellow and is .5 that they will be green. In a large number of such crosses one would expect about half the resulting peas to be yellow, the remainder to be green. In crosses between (y, g) and (y, g) peas, what proportion would be expected to be yellow? What proportion of the yellow peas would be expected to be (y, y)? *68 Peas may be smooth or wrinkled, and this is a simple Mendelian character. Smooth is dominant to wrinkled so that (s, s) and (s, w) peas are smooth while (w, w) peas are wrinkled. If (y, g) (s, w) peas are crossed with (g, g) (w, w) peas, what are the possible outcomes, and what are their associated probabilities? For the (y, g) (s, w) by (g, g) (s, w) cross? For the (y, g) (s, w) by (y, g) (s, w) cross? 69 Prove the two unproven parts of Theorem 32. 70 A supplier of a certain testing device claims that his device has high reliability inasmuch as P[A IB] = P[A IB] = .95, where A = {device indicates component is faulty} and B = {component is faulty}. You hope to use the device to locate the faulty components in a large batch of components of which 5 percent are faulty. (a) What is P[B IA]? (b) Suppose you want p[BIA] =.9. Let p=P[AIB]=P[AIB]. How large does p have to be?

II
RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

1 INTRODUCTION AND SUMMARY

The purpose of this chapter is to introduce the concepts of random variable, distribution and density functions, and expectation. It is primarily a "definitionsand-their-understanding" chapter; although some other results are given as well. The definitions of random variable and cumulative distribution function are given in Sec. 2, and the definitjons of density functions are given in Sec. 3. These definitions are easily stated since each is just a particular function. The cumulative distribution function exists and is defined for each random variable; whereas, a density function is defined only for particular random variables. Expectations of functions of random ,-ariables are the underlying concept of all of Sec. 4. This concept is introduced by considering two particular, yet extremely important, expectations. These two are the mean and variance, defined in Subsecs. 4.1 and 4.2, respectively. Subsection 4.3 is devoted to the definition and properties of expectation of a function of a random variable. A very important result in the chapter appears in Subsec. 4.4 as the Chebyshev inequality and a generalitation thereof. It is nice to be able to attain so famous a result so soon and with so little weaponry. The Jensen inequality is given in

52

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS. AND EXPECTATION

II

Subsec. 4.5. Moments and moment generating functions, which are expectations of particular functions, are considered in the final subsection. One major unproven result, that of the uniqueness of the moment generating function, is given there. Also included is a brief discussion of some measures, of some characteristics, such as location and dispersion, of distribution or density functions. This chapter provides an introduction to the language of distribution theory. Only the univariate case is considered; the bivariate and multivariate cases will be considered in Chap. IV. It serves as a preface to, or even as a companion to, Chap. III, where a number of parametric families of distribution functions is presented. Chapter III gives many examples of the concepts defined in Chap. II.

RANDOM VARIABLE AND CUMULATIVE DISTRIBUTION FUNCTION

2.1

Introduction

In Chap. I we defined what we meant by a probability space, which we denoted by the triplet (n, d, P[ . ]). We started with a conceptual random experiment; we called the totality of possible outcomes of this experiment the sample space and denoted it by n. d was used to denote a collection of subsets, called events, of the sample space. Finally our probability function P[ . ] was a set function having domain d and counterdomain the interval [0, 1]. Our object was, and still is, to assess probabilities of events. In other words, we want to model our random experiment So as to be able to give values to the probabilities of events. The notion of random variable, to be defined presently, will be used to describe events, and a cumulative distribution function will be used to give the probabilities of certain events defined in terms of random variables; so both concepts will assist us in defining probabilities of events, our goal. . One advantage that a cumulative distribution function will have over its counterpart, the probability function (they both give probabilities of events), is that it is a function with domain the real line and counterdomain the interval [0, 1]. Thus we will be able to graph it. It will become a convenient tool in modeling random experiments. In fact, We will often model a random experiment by assuming certain things about a random variable and its distribution function and in so doing completely bypass describing the probability space.

RANDOM VARIABLE AND CUMULATIVE DISTRIBUTION FUNCTION

53

2.2

Definitions

We commence by defining a random variable.

Definition 1 Random Variable For a given probability space (n, .91,

P[ . ]), a random variable, denoted by X or X(), is a function with domain nand counterdomain the real line. The function X( . ) must be such that the set A r , defined by Ar = {w: X(w) < r}, belongs to .91 for-every real number r. IIII

If one thinks in terms of a random experiment, n is the totality of outcomes of that random experiment, and the function, or random variable, X( . ) with domain n makes some real n umber correspond to each outcome of the experiment. That is the important part of our definition. The fact that we also require the collection of w's for which X(w) < rto be an event (i.e., an element of d) tor each real number r is not much of a restriction for our purposes 'since our intention is to use the notion of random variable only in descri.bing events. -.We will seldom be interested in a random variable per se; rather we will be interested in events defined in terms of random variables. One might note that the P[ . ] of our probability space (n, .91, P[ . ]) is not used in our definition. The use of words" random" and" variable" in the above definition is unfortunate since their use cannot be convincingly justified. The expression "random variable" is a misnomer that has gained such widespread use that it would be foolish for us to try to rename it. In our definition we denoted a random variable by either X( ) or X. Although X( . ) is a more complete notation, one that emphasizes that a random variable is a function, we will usually use the shorter notation of X. For manyexperiments, there is a need to define more than one random variable; hence further notations are necessary. We will try to use capital Latin letters with or without affixes from near the end of the alphabet to denote random variables. Also, we use the corresponding small letter to denote a value of the random variable.

EXAMPLE 1 Consider the experiment of tossing a single coin. Let the random variable X denote the number of heads. n = {head, tail}, and X(w) = 1 if w = head, and X(w) = 0 if w = tail; so, the random variable X associates a real number with each outcome of the experiment. We called X a random variable so mathematically speaking we should show

54

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

II

2d dice

5
4 3

6 1st dice

2
1

,.

FIGURE 1

that it satisfies the definition; that is, we should show that {w: X(w) < r} belongs to d for every real number r. d consists of the four subsets: 4>, {head}, {tail}, and n. Now, if r < 0, {w: X(w) < r} = 4>; and if <r < I, {w: X(w) < r} = {tail}; and if r > 1, {w: X(w) < r} = n = {head, tail}. Hence, for each r the set {w: X(w) < r} belongs to d; so X( . ) is arandom variable. IIII

EXAMPLE 2 Consider the experiment of tossing two dice. n can be described by the 36 points displayed in Fig. 1. n = {(i, j): i = I , ... , 6 and j = 1, ... , 6}. Several random variables can be defined; for instance, let X denote the sum of the upturned faces; so X(w) = i + j if w = (i, j). Also, let Y denote the absolute difference between the upturned faces; then Y(w) = ]i - j] if w = (i, j). It can be shown that both X and Yare random variables. We see that X can take on the values 2, 3, ... , 12 and Y can take on the values 0, 1, ... , 5. IIII In both of the above examples we described the random variables in terms of the random experiment rather than in specifying their functional form; such will usually be the case.

Definition2 Cumulative distribution function Thecumulativedistribution function of a random variable X, denoted by Fx ('), is defined to be that function with domain the real line and counterdomain the interval

RANDOM VARIABLE AND CUMULATIVE DISTRIBUTION FUNCTION

55

[0, 1] which satisfies Fx(x) numberx.

P[X

x]

P[{w: X(w) < x}] for every real

I1II

A cumulative distribution function is uniquely defined for each random variable. If it is known, it can be used to find probabilities of events defined in terms of its corresponding random variable. (One might note that it is in this definition that we use the requirement that {w: X(w) < r} belong to d for every real r which appears in our definition of random variable X.) Note that different random variables can have the same cumulative distribution function. See Example 4 below. The use of each of the three words in the expression" cumulative distribution function" is justifiable. A cumulative distribution function is first of all a/unction; it is a distribution function inasmuch as it tells us how the values of the random variable are distributed, and it is a cumulative distribution function since it gives the distribution of values in cumulative form. Many writers omit the word "cumulative" in this definition. Examples and properties of cumulative distribution functions follow.

EXAMPLE 3 Consider again the experiment of tossing a single coin. Assume that the coin is fair. Let X denote the number of heads. Then, if x <0 if 0 x < 1 if 1 x. Or Fx(x)

1/ [0, .)(x) + 1[1, oo)(x) in our indicator function notation. III1

EXAMPLE 4 In the experiment of tossing two fair dice, let Y denote the absolute difference. The cumulative distribution of Y, Fy( . ), is sketched in Fig. 2. Also, let X k denote the value on the upturned face of the kth die for k 1, 2. Xl and X 2 are different random variables, yet both have the same cumulative distribution function, which is FXk(x) =
5

l~

6Ip, i+ I)(X) + 1[6, oo)(X) and is sketched in Fig. 3.

IIII

Careful scrutiny of the definition and above examples might indicate the following properties of any cumulative distribution function Fx( . ).

S6

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

II

Fr(y)

34/36 30/36 24/36 16/36 6/36

------------

T
I
I I
!

FIGURE 2

L. 2

I 3

I 4

.. y

Properties of a Cumulative Distribution Function F x (')

(i) (ii)

Fx(

ex))

= lim Fx(x}
x ..... -oc

0, and Fx(

ex))

lim Fx(x) = 1.
x-+oo

Fx (') is a monotone, nondecreasing function; that is, Fx(a) < Fx(b} for a < b.
(iii)

Fx (') is continuous from the right; that is,

lim Fx(x o <h ..... O

+ h) =

Fx(x}.

Except for Oi), we will not prove these properties. Note that the event {w: X(w) b} {X < b} {X < a} u {a < X b} and {X a} n {a < X <b} = 4>; hence, Fx(b} P[X < b] = P[X < a] + P[a < X b] P[X < a] = Fx(a) which proves (ii). Property (iii), the continuity of Fx( . ) from the right, results from our defining Fx(x) to be P[X < xl lf we had defined. as some authors do, Fx{x) to b~ P[X < x], then Fx( . ) would have been continuous from the left.

Definition 3 Cumulative distribution function Any function F( . } with domain the real line and counterdomain the interval [0, J] satisfying the above three properties is defined to be a cumulative distribution/unction.

IIII
This definition allows us to use the term" cumulative distribution func~ tion" without mentioning random variable. After defining what is meant by continuous and discrete random variables in the first two subsections of the next section, we will give another property that cumulative distribution functions possess, the property of decomposition into three parts.

DENSITY FUNCTIONS

57

FJGURE 3

The cumulative distribution functions defined here are univariate; the introduction of bivariate and multivariate cumulative distribution functions will be deferred until Chap. IV.

DENSITY FUNCTIONS

Random variable and the cumulative distribution function of a random variable have been defined. The cumulative distribution function described the distribution of values of the random variable. For two distinct classes of random variables, the distribution of values can be described more simply by using density functions. These two classes, distinguished by the words "discrete" and" continuous," are considered in the next two subsections.

3.1

Discrete Random Variables Definition 4 Discrete random variable A random variable X will be
defined to be discrete if the range of X is countable. If a random variable X is discrete, then its corresponding cumulative distribution function Fx( . ) will be defined to be discrete. IIII

By the range of X being countable we mean that there exists a finite or denumerable set of real numbers, say XI' X 2 , X 3 , .. , such that X takes on values only in that set. If X is discrete with distinct values x I, X2, .. , x"' ... , then 0= {w: X(w) = x,,} = {X = x,,}, and {X = xJ n {X = x) = 4> for i # j;

U
"

U
"

hence 1 = prO]

= L P[X =
"

x,,] by the third axiom of probability.

58

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

II

Definition 5 Discrete density function of a discrete random variable If X is a discrete random variable with distinct values X h Xl' . , X n , , then the function, denoted by fx( . ) and defined by if X if x =I:
Xj' Xj

j = I, 2, ... , n, ...

(1)

is defined to be the discrete density function of X.

IIII

The values of a discrete random variable are often called mass points; and, fx(xj) denotes the mass associated with the mass point x j Probability mass function, discrete frequency function, and probability function are other terms used in place of discrete density function. Also, the notation px( .) is sometimes used intead of fx( . ) for discrete density functions. fx(') is a function with domain the real line and counterdomain the interval [0, 1]. If we use the indicator function,

fx(x)
n

L P[X = xn]/{x,,}(x),
I

00

(2)

Theorem 1 Let Xbe a discrete random variable. fromfx('), and vice versa.
PROOF

Fx (') can be obtained

Denote the mass points of Xby Xl, Xl, Suppose fx(') is given; then Fx(x) = fx(x). Conversely, suppose Fx(') is given;
{j: xr';x}

then fx(xj) = Fx(xj ) -

lim Fx(xj - h); hence fx(x) can be found for

O<h ..... O

each mass point X j; however, fx(x) determined for all real numbers.

0 for X =I: X J' j

1, 2, ... , so fx(x) is

IIII

EXAMPLE 5 To illustrate what is meant in Theorem 1, consider the experiment of tossing a single die. Let X denote the number of spots on the upper face:
fx( x)

(~) 1(1.2 . 6)(X)'

and
Fx(X) =

t:= I

L (i16)/[i,

i+ 1)(x)

+ 1[6. oo)(X).

DENSITY FUNCTIONS

59

-h
0
1

f6
4

-h

-A

36

16
10

io
11

12

FIGURE 4

According to Theorem 1, for given fx('), Fx(x) can be found for any x; for instance, if x = 2.5,

And, if Fx (') is gjven, fx(x) can be found for any x. x = 3,

fx(3)

For example, for

= Fx(3)

o~~o Fx(3 -

h)

W -W =~.

IIII

The cumulative distribution function of a discrete random variable has steps at the mass points; that is, at the mass point x j ' Fx (') has a step of size fx{x J) , and Fx {') is flat between mass points.

EXAMPLE" 6 Consider the experiment of tossing two dice. Let X denote the total of the upturned faces. The mass points of X are 2, 3, ... , 12. lx(') is sketched in Fig. 4. Let Y denote the absolute difference of the upturned faces; thenfy(') is given in tabular form by
Y

0
36
6

I
J6
10

2
36
8

3
36
6

4
36
4

5
..L
36

fly)

11I1

The discrete density function tells us how likely or probable each of the values of a discrete random variable is. It also enables one to calculate the probability of events described in terms of the discrete random variable X. For example, let X have mass points XI' x 2 , , X n , ; then P[a < X b] = L !x(Xj) for a < b.
j:{ a< XJ :;;b}

60

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

II

Definition 6 Discrete density function Any function f( . ) with domain

the real line and counterdomain [0, 1] is defined to be a discrete density function if for some countable set Xl' x 2 , , X n , ,

(0 f(x) > 0 for j

(ii) f(x) = 0 for
(iii)
X

= 1, 2, ....

#=

Xj;j

= 1,2, ....
XI'

L f(x)) = 1,

where the summation is over the points

x2 ,

"

Xn ,

IIII

This definition allows us to speak of discrete density functions without reference to some random variable. Hence we can talk about properties that a density function might have without referring to a random variable.

3.2

Continuous Random Variables Definition 7 Continuous random variable A random variable X is

called continuous if there exists a functionfx(') such that Fx(x) =

J fx(u)du
-00

for every real number x. The cumulative distribution function Fx( .) of a continuous random variable X is called absolutely continuous. IIII

Definition 8 Probability density function of a continuous random variable

If X is a continuous random variable, the function fx(')
In

Fx(x) =

fx(u) du is called the probability density function of X.

I111

-00

Other names that are used instead of probability density function include density function, continuous density function, and integrating density function. , Note that strictly speaking the probability density function fx(') of a random variable X is not uniquely defined. All that the definition requires is that the integral of fx(') gives Fx(x) for every x, and more than one function fx(') may satisfy such requirement. For example, suppose Fx(x) = x/[o, I)(x) +
1[I.oo)(x); then fx(u) =
1(0,

I)(u) satisfies Fx(x) =

J fx(u)
-00

du for every x, and

::=: 1(0. tlu)

so fx(') is a probability density function of X. However fx(u)

69/(t}(u)

+ l<t, I)(u)

also satisfies Fx(x) =

fx(u) duo

(The idea is that ,if the

-00

value of a function is changed at only a "few" points, then its integral is unchanged.) In practice a unique choice of fx(') is often dictated by continuity' considerations and for this reason we will usually allow ourselves the liberty of

DENSITY fUNCTIONS

61

speaking of the probability density when in fact a probability density is more correct. One should point out that the word "continuous" in "continuous random variable" is not used in its usual sense. Although a random variable is a function and the notion of a continuous function is fairly well established in mathematics, .. continuous" here is not used in that usual mathematical sense. In fact it is not clear in what sense it is used. Two possible justifications do come to mind. In contrasting discrete random variables with continuous random variables, one notes that a discrete random variable takes on a finite or denumerable set of values whereas a continuous random variable takes. on a nondenumerable set of values. Possibly it is the connection between " nondenumerable" and continuum" that justifies use of the word "continuous." All the continuous random variables that we shall encounter will take on a continuum of values .. The second justification arises when one notes that the absolute continuity of the cumulative distribution function is the regular mathematical definition of an absolutely continuous function (in words, a function is called absolutely continuous if it can be written as the integral of its derivative); the "continuous," then, in a corresponding continuous random variable could be considered just an abbreviation of" absolutely continuous."
U

Theorem 2 Let X be a continuous random variable. Then Fx (') can

be obtained from an/ x {'), and vice versa.
PROOF

If X is a continuous random variable and an /x( .) is given,

then Fx(x) is obtained by integrating/ x ('); that is, Fx(x) =

:x

ix(u) duo

On

-00

the other hand, if Fx(') is given, then an/x(x) can be obtained by differentiation; that is, /x(x) = dFx(x)/dx for those points x for which Fx(x) is differentiable. II11 The notations for discrete density function and probability density function are the same, yet they have quite different interpretations. For discrete random variables /x(x) = P[X = xl, which is not true for continuous random variables. For continuous random variables,

f x (x ) -

dFx(x) _ l' - 1m Fx(x dx 4:x-O

+ ~x) -

Fx(x - ~x) . 2~x '

hence fx(x)2~x ~ Fx(x + ~x) - Fx(x - ~x) = P[x - Ax < X < x + ~xl; that is, the probability that X is in a small interval containing the value x is approximately equal to /x(x) times the width of the interval. For discrete random

62

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

II

variables fx(') is a function with domain the real line and counterdomain the interval [0,.1]; whereas, for continuous random variables fx(') is a function with domain the real line and counterdomain the infinite interval [O~ (0) .

. Remark We will use the term" density function" without the modifier
of" discrete" or "probability" to represent either kind of density.

1111

EXAMPLE 7 Let X be the random variable representing the length of a telephone conversation. One could model this experiment by assuming that the distribution of X is given by Fx(x) = (1 - e AX)/[O.oo,(x), where l is some positive number. The corresponding probability density function would be given by fx(x) = le-AX/[O, oo)(x). If we assume that telephone conversations are measured in minutes, P[5 < X < 10] = le-xAdx = e- SA - e- 10A = e- 1 - e- 2 ~.23forl = t,orP[5 < X < 10] = P[X < 10] - P[X::; 5] = (I - e AI0) - (I - e A.s) = e- 1 - e- 2 for

n
=

/1//

The probability density function is used to calculate the probability of events defined in terms of the corresponding continuous random variable X. For ex~mple, P[a < X < b) = S!fx(x) dx for a < b. Definition 9 Probability density function Any function f( . ) with domain the real line and counterdomain [0, (0) is defined to be a probability density function if and only if

(0 f(x) >
IX)

for all x.
1.

(ii)

J f(x) dx =
-IX)

III/

With this definition we can speak of probability density functions without reference to random variables. We might note that a probability density function of a continuous random variable as defined in Definition 8 does indeed possess the two properties in the above definition.

3.3

Other Random Variables

Not all random variables are either continuous or discrete, or not all cumulative distribution functions are either absolutely continuous or discrete.

DENSITY FUNCTIONS

63

______

______

~-

___________

--~----

__ x
~

FIGURE 5

EXAMPLE 8 Consider the experiment of recording the delay that a motorist encounters at a one..way traffic stop sign. Let X be the random variable that represents the delay that the motorist experiences after making the required stop. There is a certain probability that there will be no opposing traffic so that the motorist will be able to proceed with no delay.' On the other hand, if the motorist has to wait, he may have to wait for any of a continuum of possible times. This experiment could be modeled by assuming that X has a cumulative distribution function given by Fx{x) = (1 - pe AX)/[O,OO)(x). This Fx(x) has a jump of I - p at x = 0 but is continuous for x > O. See Fig. 5. IIII Many practical examples of cumulative distribution functions that are partly discrete and partly absolutely continuous can be given. Yet there are still other types of cumulative distribution functions. There are continuous cumulative distribution functions, called singular continuous, whose derivative is 0 at alm<?st all points. We will not consider such distribution functions other than to note the following result. Decomposition of a cumulative distribution function distribution function F(x) may be represented in the form

Any cumulative

where PI > 0, i = 1,2,3.

(3)

i= 1

L Pi =

I, and F d ( ), F ac( ), and pc ( . ) are each cumulative distribution functions

with F d (.) discrete, Fac{.) absolutely continuous, and P SC( . ) singular continuous. Cumulative distributions studied in this book will have at most a discrete part and an absolutely continuous part; that is, the P3 in Eq. (3) will always be 0 for the F(') that we will study.

64'

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

II

EXAMPLE 9 To illustrate how the decomposition of a cumulative distribution function can be implemented, consider Fx(x) = (l - pe-Ax)/[o, oo)(x) as in Example 8. Fx(x) = (l - p)Fd(X) + ppac(x), where Fd(X) = 1[0, OO)(x) and Fac(x) = (1 - e-Aj/[o. OO)(x). Note that Fx(x) = (l - p)Fd(x) + pFac(x) = (l - p)/[o. OO)(x) + p(l - e-Ax)/[o, oo)(x) = (l - pe-AX)I[o, oo)(x).

IIII
A density function corresponding to a cumulative distribution that is partly discrete and partly absolutely continuous could be defined as follows: If F(x) = (l - p)Fd(X) + pFac(x), where 0 < p < I and F d(.) and F ac (.) are, respectively, discrete and absolutely continuous cumulative distribution functions, let the density function f(x) corresponding to F(x) be defined by f(x) = (l - p)~(x) + pfac(x), where r(') is the discrete density function corresponding to F d (.) andfac(.) is the probability density function corresponding to F ac (.). Such a density function would require careful interpretation; so when considering cumulative distribution functions that are partly discrete and partly continuous, we will tend to work with the cumulative distribution function itself rather than with a density function.

Remark In future chapters we will frequently have to state that a random variable has a certain distribution. We will make such a statement by giving either the cumulative distribution function or the density IIII function of the random variable of interest.

EXPECTATIONS AND MOMENTS

An extremely useful concept in problems involving random variables or distributions is that of expectation. The subsections of this section give definitions and results regarding expectations.

4.1

Mean Definition 10 Mean Let X be a random variable. denoted by /-lx or G[X], is defined by:
(i)

The mean of X,
(4)

if X is discrete with mass points

Xl'

x2 ,

.. ,

xj

' ...

EXPECTATIONS AND MOMENTS

65

(ii)

G[X] =

f- oo xfx (x)dx
00

(5)

if X is continuous with probability density functionfx(x). (iii)

G[X] =

fo

00

[1 - Fx(x)] dx -

fO _ooFx(X) dx

(6)

for an arbitrary random variable X.

IIII

In 0), G[X] is defined to be the indicated series provided that the series is absolutely convergent; otherwise, we say that the mean does not exist. And in (ii), G[X] is defined to be the indicated integral if the integral exists; otherwise, we say that the mean does not exist. Final!y, in (iii), we require that both integrals be finite for the existence of G[X]. Note what the definition says: In xjfx(xj), the summand is thejth value

L
j

of the random variable X multiplied by the probability that X equals that jth value, and then the summation is overall values. So G[X] is an" average" of the values that the random variable takes on, where each value is weighted by the probability that the random variable is equal to that value. Values that are more probable receive more weight. The same is true in integral form in (ii). There the value x is multiplied by the approximate probabjlity that X equals the value x, namely fx(x) dx, and then integrated over all values. Several remarks are in order.

Remark In the definition of a mean of a random variable, only density functions [in (0 and (ii)] or distribution functions [in (iii)] were used; hence we have really defined the mean for these functions without reference to random variables. We then call the defined mean the mean of the cumulative distribution function or of the appropriate density function. Hence, we can and will speak of the mean of a distribution or density IIII function as well as the mean of a random variable.

Remark G[X] is the center of gravity (or centroid) of the unit mass that is determined by the density function of X. So the mean of X is a measure of where the values of the random variable X are" centered." Other measures of "location" or "center" of a random variable or its corresponding density are given in Subsec. 4.6. 7Tn '

66

RANDOM VARIABLES; DISTRIBUTION FUNCI10NS; AND EXPECTATION

II

Remark (iii) of the definition is for all random variables; whereas, (i) is for discrete random variables, and (ii) is for continuous random variables. Of course, tC[X] could have been defined by just giving (iii). The reason for including (i) and (ii) is that they are more intuitive for their respective cases. It can be proved, although we will not do it, that (i) follows from (iii) in the case of discrete random variables and (ii) follows from (iii) in the case of continuous random variables. Our main use of (iii) will be in finding the mean of a random variable X that is neither IIII discrete nor continuous. See Example 12 below. EXAMPLE 10 Consider the experiment of tossing two dice. Let X denote the total of the two dice and Y their absolute difference. The discrete density functions for X and Yare given in Example 6.
5

tC[Y] = LYjfY(yj) = L ifrCi)

i=O

o 366 + 1 ~~
7.

2-1L+3~+4--"'+5--L=70 36 36 36 36 36

12

tC[X] =

1=2

L ifx(i) =

Note that tC[ Y] is not one of the possible values of Y.

IIII

EXAMPLE 11 Let X be a continuous random variable with probability density functionfx(x) = le- Ax/[o. oolx).
tC[X]

oo

xfx(x) dx

1 = foo xle- Ax dx = ,.
0
I\.

-00

The corresponding cumulative distribution function is

IIII
EXAMPLE 12 Let X be a random variable with cumulative distribution function given by Fx<.x) = (1 - pe-A,/[o. oolx); then
tC[X]

= f0 [1 - Fx(x)] dx -

oo

fO
-00

Fx(x) dx

foo P AX 0 pedx =;:.

Here, we have used Eq. (6) to find the mean of a random variable that is partly discrete and partly continuous. IIII

EXPECTATIONS AND MOMENTS

67

EXAMPLE 13 Let X be a random variable with probability density function given by fx(x) = x- 2 /[1,(0)(x); then
S[X] =

oo

x 2" = lIm loge b =

X
b ...... oo

dx

00,

so we say that S[X] does not exist.' We might also say that the mean of X is infinite since it is clear here that the integral that defines the mean is
~~

4.2 Variance
The mean of a random variable X, defined in the previous subsection, was a measure of central location of the density of X. The variance of a random variable X will be a measure of the spread or dispersion of the density of X. Definition 11 Variance Let X be a random variable, and let Ilx be S[x]. The variance of X, denoted by O'i or var [X], is defined by
(i)

var [X] =
Xh

L (Xj j

Ilx)2fx(xj)
'

(7)

if X is discrete with mass points (ii) var [X]

x2
00 00

, .. ,

xj

f_

(x - Ilx)2fx(x) dx

(8)

if X is continuous with probability density functionfx(x). (iii) var [X] =

fo

00

2x[1 - Fx(x)

+ Fx( -

x)] dx - Ili

(9)

for an arbitrary random variable X.

IIII

The variances are defined only if the series in (i) is convergent or if the integrals in (ii) and (iii) exist. Again, the variance of a random variable is defined in terms of the density function or cumulative distribution function of the random variable; hence variance could be defined in terms of these functions without reference to a random variable. Note what the definition says: In (i), the square of the difference between the jth value of the random variable X and the mean of X is multiplied by the probability that X equals the jth value, and then these terms are summed. More weight is assigned to the more probable squared differences. A similar comment applies for (ii). Variance is a measure of spread since if the values of a random variable X tend to be far from their mean, the variance of X will be larger than the variance of a comparable random variable Y whose values tend to be near their mean. It is clear from (i) and (ii) and true for (Hi) that variance is nonnegative. We saw that a mean was the center of gravity of a

68

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

II

density; similarly (for those readers familiar with elementary physics or mechanics), variance represents the moment of inertia of the same density with respect to a perpendicular axis through the center of gravity. Definition 12 Standard deviation If X is a random variable, the

standard deviation of X, denoted by ax, is defined as

+ Jvar [Xl.

IIII

The standard deviation of a random variable, like the variance, is a measure of the spread or dispersion of the values of the random variable. In many applications it is preferable to the variance as such a measure since it will have the same measurement units as the random variable itself.

EXAMPLE 14 Let X be the total of the two dice in the experiment of tossing
two dice. var [X] = L(Xj - JlX)2/X(Xj)

= (2 - 7)2l6 + (3 - 7) 2l6 + (4 - 7)2 336 + (5 - 7)2 346

+ (6 - 7)2 356 + (7 + (10 - 7)2 /6 + (11

66 + (8 - 7)2 356 + (9 - 7)2 3 4 7)2 3 6 1 - 7) 2l6 + (12 _7)2 3 6 = 2l6 .

IIII

EXAMPLE 15 Let X be a random variable with probability density given by /x(x) = ).e-;,xI[o, oo)(x); then
Var [X] = foo (x - Jlx)2fx(x) dx
=

r
-

00

(x - W.<e-

AX

dx

1 - A?'

IIII

EXAMPLE 16 Let X be a random variable with cumulative distribution given by Fx(x) = (1 - pe-;'X)I[o. OO)(x); then
Var [X] = foo 2x[1 - F(x) + F( -x)] dx =

r
o

Jli

2xpe- AX dx -

(~)'
).2 .

= 2

).2

~_

(E) ).

2 _

p(2 - p)

IIII

EXPECTATIONS AND MOMENTS

69

4.3

Expected Value of a Function of a Random Variable

We defined the expectation of an arbitrary random variable X, called the mean of X, in Subsec. 4.1. In this subsection, we will define the expectation of a function of a random variable for discrete or continuous random variables. Definition a function expectation X, denoted
(i)

Let X be a random variable and g(.) be with both domain and counterdomain the real line. The or expected value of the function g(') of the random variable by ..&'[g(X)], is defined by:
..&'[g(X)] =
Xb

13 Expectation

L g(xj)fx(x j)
j

(10)

if X is discrete with mass points series is absolutely convergent). (ii)

x2

, .. , Xj' ..

(provided this

..&'[g(X)] =

f_ oog(x)/x(x) dx
00

(11)

if X is continuous with probability density function fx(x) (provided S~oolg(x)lfx(x) dx < (0).* IIII Expectation or expected value is not really a very good name since it is not necessarily what you "expect." For example, the expected value of a discrete random variable is not necessarily one of the possible values of the discrete random variable, in which case, you would not" expect" to get the expected value. A better name might be " average value" rather than" expected value." Since -..&'[g(X)] is defined in terms of the density function of X, it could be defined without reference to a random variable. Remark If g(x) = x, then ..&'[g(X)] = ..&'[X] is the mean of X. (x - f.lx)2, then ..&'[g(X)] = ..&'[(X - f.lX)2] = var [Xl If g(x) =

IIII

* tf[g(X)] has been defined here for random variables that are either discrete or continuous; it can be defined for other random variables as well. For the reader who is familiar with the Stieltjes integral, C[g(X)J is defined as the Stieltjes integral J~ oog(x) dFx(x) (provided this integral exists). where F x() is the cumulative distribution function of X. If X is a random variable whose cumulative distribution fUnction is partly discrete and partly continuous. then (according to Subsec. 3.3) Fx(x) = (l - p)r(x) + pPC(x) for some 0 < p < 1. Now tf[g(X)J can be defined to be tf[g(X)J = (1- p) g(X))fd(X)) + p J~ rz;)g(x)fac(x) dx, where fd(.) is the discrete density function corresponding to Fd(.) and r C ( . ) is the probability density function corresponding to F ac (.).

2:

70

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

II

Theorem 3 Below are properties of expected value:

0) S[c] = c for a constant c.

(ii) S[cg{X)]

= cS[g(X)] for a constant c.

=

(iii) S[Clg1(X) + C2g2(X)] (iv) S[gl(X)]

PROOF

ClS [gl{X)]

+ C2S[g2(X)].

S[g2{X)] if gl(X) < g2(X) for an x.

Assume X is continuous.

To prove (i), take g(x) = c, then

S[g{X)] = S[c] S[cg(X)] =

= foo
-00

cfx(x) dx = c

foo fx(x) dx = c.
-00

f_

00 00

cg(x)/x(x) dx = c

f_

00 00

g(x)/x{x) dx = cS[g(X)],

which proves (ii).

(iii) is given by

= Cl

f- oo gl(X)/X(x) dx + Cz f- oo gix)/x(x) dx
+ Cz S[g2(X)]'

00

00

= C 1 S[gl(X)]

Finally,

which gives (iv). Similar proofs could be presented for the discrete random variable
~~.

ffll

Theorem 4 If X is a random variable, var [X] S[X2] - (S[XD 2 provided 8[X2] exists.
PROOF

= S[(X -

S[XD2] =

CNe first note that if S[X2] exists, then S[X] exists.)* By our definitions of variance and S[g(X)], it follows that var [X] = S[eX - S[X])2]. Now S[(X - S[X])2] = S[X2 - 2XS[X] + (S[XD2] = S[X2]- 2(S[X])2 + {S[X])2 = S[X2] - {S[X])2. 1//1 The above theorem provides us with two methods of calculating a variance, namely S[(X - IlX)l] or S[X2] Ili. Note that both methods require Ilx.

* Here and in the future We are not going to concern ourselves with checking existence.

EXPECTATIONS AND MOMENTS

71

8[g(X)] is used in each of the following three subsections. In Subsec. 4.4 and 4.5 two inequalities involving 8[g(X)] are given. Definitions and examples of 8[g(X)1 for particular functions g(') are given in Subsec. 4.6.

4.4 Chebyshev Inequality

Theorem 5 Let X be a random variable and g(' ) a nonnegative function with domain the rea11ine; then
c

P[g(X) > k] < 8[g!X)]

PROOF

-.::.{.
~y.

for every k > O.

(12)

" 1I'~t.:.

Assume that X is a continuous random variable with probability density functionfx('); then
8[g(X)] =

f>

-00

f >f
+

{x:g(x)<t}

f g(x)/x(x) dx > f
g(x)/x(x) dx

g(x)/x<x) dx g(x)/x{x) dx

~g~)~~

{x: g(x)~t}

k/X<x) dx = kP[g(X)

(x: .(x)~"}

~ k].

Divide by k, and the result follows.

A similar proof holds for X discrete.

1111
Corollary Chebyshev inequality If X is a random variable with finite vanance, for every r > O. (13)
PROOF

Takeg(x) = (x - Px)2 and k = r 2 ui in Eq.(12) of Theorem 5.

1/11
Remark If X is a random variable with finite variance,
1 P[I X - Pxl < rux] ~ 1 - "2'

(14)

which is just a rewriting of Eq. (13).

1/1/

The Chebyshev inequality is used in various ways. We will use it later to prove the law of large numbers. No~e what Eq. (14) says:
P[Px - rux < X < Px

+ rux] >

I - ..;.;

72

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS. AND EXPECTATION

II

that is. the probability that X falls within ru x units of Jlx is greater than or equal to I - llr2. For r = 2, one gets P[Jtx - 2ux < X < Jlx + 2u x ] > t, or for any random variable X having finite variance at least three-fourths of the mass of X falls within two standard deviations of its mean. Ordinarily, to calculate the probability of an event described in terms of a random variable X, the distribution or density of X is needed; the Chebyshev inequality gives a bound, which does not depend on the distribution of X, for the pro bability of particular events described in terms of a random variable and its mean and variance.

4.5

Jensen Inequality Definition 14 Convex function A continuous function g( . ) with domain and counterdomain the real line is called convex if for every Xo on the real line, there exists a line which goes through the point (xo, 'g(xo)) and lies on or under the graph of the function g('). IIII Theorem 6 Jensen inequality Let X be a random variable with mean tS'[X], and let g(') be a convex function; then tS'[g(X)] ~ g(tS'[X]). Since g(x) is continuous and convex, there exists a line, say /(x) = a + bx, satisfying /(x) = a + bx < g(x) and /(tS'[X]) = g(tS'[X]). /(x) is a line given by the definition of continuous and convex that goes through the point (tS'[X], g(tS'[X])). Note that tS'[/(X)] = tS'[(a + bX)] = a + btS'[X] = /(tS'[X]); hence g(tS'[X]) = /(tS'[X]) = tS'[/(X)] < tS'[g(X)] [using property {iv} of expected values (see Theorem 3) for the last inequality].
PROOF

IIII
The Jensen inequality can be used to prove the Rao-Blackwell theorem to appear in Chap. VII. We point out that, in general, tS'[g(X)] - g(tS'[X]); for example, note that g(x) = x 2 is convex; hence tS'[X2] > (tS'[X])2, which says that the variance of X, which is tS'[X2] - (tS'[X]) 2 , is nonnegative.

4.6

Moments and Moment Generating Functions

The moments (or raw moments) of a random variable or of a distribution are the expectations of the powers of the random variable which has the given distribution.

EXPECTATIONS AND MOMENTS

73

Definition 15 Moments If X is a random variable, the rth moment of X, usually denoted by Il;, is defined as
Il; = G[X']

(15)

if the expectation exists. Note that

Il~

IIII
the mean of X.

= G[X]

= Ilx,

Definition 16 Central moments If X is a random variable, the rth central moment of X about a is defined as G[(X - aYl If a = Ilx, we have the rth central moment of X about Ilx, denoted by Il" which is
Il,

= G[(X -

IlxYl

(16)

IIII
Note that III = G[(X - Ilx)] = 0 and 112 = G[(X - IlX)2], the variance of X. Also, note that all odd moments of X about Ilx are 0 if the density function of X is symmetrical about Ilx, provided such moments exist. In the ensuing few paragraphs we will comment on how the first four moments of a random variable or density are used as measures of various , .characteristics of the corresponding density. For some of these characteristics, '.' other measures can be defined in terms of quantiles. Definition 17 Quantile The qth quantile of a random variable X or of its corresponding distribution is denoted by q and is defined as the smallest number satisfying Fx(e) > q. IIII

If X is a continuous random variable, then the qth quantile of X is given as the smallest number satisfying Fx(e) = q. See Fig. 6.

Definition 18 Median The median of a random variable X, denoted by medx , med (X), or (so, is the .5th quantile. IIII Remark In some texts the median of X is alternatively defined as any number, say med (X), satisfying P[X ~ med (X)] > t and P[X > med (X)]
~

IIII

If X is a continuous random variable, then the median of X satisfies

f
-

mea (X)

fx(x) dx
c)

=t =

c)

fx(x) dx;

med (X)

74

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

II

Fx(x)

1.0
1

Fx(x)

.75

.50
.25

FIGURE 6

So the median of X is any number that has half the mass of X to its right and the other half to its left, which justifies use of the word median." We have already mentioned that8[X], the first moment, locates the" center" of the density of X. The median of X is also used to indicate a central location of the density of X. A third measure of location of the density of X, though not necessarily a measure of central location, is the mode of X, which is defined as that point (if such a point exists) at whkh fx(') attains its maximum. Other measures of location [for example, t('.25 + '.75)] could be devised, but three, mean, median, and mode, are the ones commonly used. We previously mentioned that the second moment about the mean, the variance of a distribution, measures the spread or dispersion of a distribution. Let us look a little further into the manner in which the variance characterizes the distribution. Suppose that It (x) and f2(x) are two densities with the same mean f.l such that
H

{-a [fleX) - f2(X)] dx > 0

p+a

(17)

for every value of a. Two such densities are illustrated in Fig. 7. It can be shown that in this case the variance ai in the first density is smaller than the

FIGURE 7

EXPECTATIONS AND MOMENTS

75

FIGURE 8

variance CT~ in the second density. We shall not take the time to prove this in detail, but the argument is roughly this: Let

g(x)
where It (x) and f2(X) satisfy Eq. (17).
co

= It (x) - f2(X) ,
dx = 0, the positive area

Since

S g(x)
- co

between g(x) and the x axis is equal to the negative area. Furthermore, in view of Eq. (17), every positive element of area g(x') dx' may be balanced by a negative element g(x") dx" in such a way that x" is further from J-l than x'. When these elements of area are multiplied by (x - J-l)2, the negative elements will be multiplied by larger factors than their corresponding positive elements (see Fig. 8); hence

JCO (x - J-l)2 g (X) dx < 0

-00

unless It (x) and f2(X) are equal. Thus it follows that ui < u~ . The converse of these statements is not true. That is, if one is told that ui < u~ , he cannot conclude that the corresponding densities satisfy Eq. (17) for all values of a; although it can be shown that Eq. (17) must be true for certain values of a. Thus the condition ui < u~ does not give one any precise information about the nature of the corresponding distributions, but it is evident that It (x) has more area near the mean thanf2(x), at least for certain intervals about the mean. We indicated above how variance is used as a measure of spread or dispersion of a distribution. Alternative measures of dispersion can be defined in terms of quantiles. For example 7S 2S , called the interquartile range, is a measure of spread. Also, p p for some -!<p < I is a possible measure of spread. The third moment J-l3 about the mean is sometimes called a measure of asymmetry, or skewness. Symmetrical distributions like those in Fig. 9 can be shown to have J-l3 = O. A curve shaped likelt(x) in Fig. 10 is said to be skewed to the left and can be shown to have a negative third moment about the mean; one shaped like f2{X) is called skewed to the right and can be shown to have a positive third moment about the mean. Actually, however, knowledge of the

e el-

e.

e.

76

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

II

--~--~------------~--------------~~-------x

FIGURE 9

third moment gives almost no clue as to the shape of the distribution, and we mention it mainly to point out that fact. Thus, for example, the density f3(x) in Fig. 10 has /13 = 0, but it is far from symmetrical. By changing the curve slightly we could give it either a positive or negative third moment. The ratio /13/(13, which is unitiess, is called the coefficient of skewness. The quantity 11 = (mean - median)/(standard deviation) provides an alternative measure of skewness. It can be proved that -1 < 11 < 1. The fourth moment about the mean is sometimes used as a measure of excess or kurtosis, which is the degree of flatness of a density near its center. Positive values of /14-/(14 - 3, called the coefficient 0/ excess or kurtosis, are sometimes used to indicate that a density is more peaked around its center than the density of a normal curve (see Subsec. 3.2 of Chap. III), and negative values are sometimes used to indicate that a density is more flat around its center than the density of a normal curve. This measure, however, suffers from the same failing as does the measure of skewness; namely, it does not always measure what it is supposed to. While a particular moment or a few of the moments may give little information about a distribution (see Fig. 11 for a sketch of two densities having the same first four moments. See Ref. 40. Also see Prob. 30 in Chap. Ill), the entire set of moments (J1~, /1~, f.L;, ...) will ordinarily determine the distri-

~X~)
FIGURE 10

__LL__

~ ~=-____________~_
__

EXPECTATIONS AND MOMENTS

77

.7
.6

.5

.4
.3
.2

.1

-2
FIGURE 11

bution exactly, and for this reason we shall have occasion to use the moments in theoretical work. In applied statistics, the first two moments are of great importance, as we shall see, but the third and higher moments are rarely useful. Ordinarily one does not know what distribution function one is working with in a practical problem, and often it makes little difference what the actual shape of the distribution is. But it is usually necessary to know at least the location of the distribution and to have some idea of its dispersion. These characteristics can be estimated by examining a sample drawn from a set of objects known to have the distribution in question. This estimation problem is probably the most important problem in applied statistics, and a large part of this book will be devoted to a study of it. We now define another kind of moment,/actorial moment.

Definition 19 Factorial moment If X is a random variable, the rth /actorial moment 0/ X is defined as (r is a positive integer):
CrX(X - 1) ... (X - r

+ I)].

(18)

I111
For some random variables (usually discrete), factorial moments are

78

RANDOM VARlABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

easier to calculate than raw moments. However the raw moments can be obtained from the factorial moments and vice versa. The moments of a density function play an important role in theoretical and applied statistics. In fact, in some cases, if all the moments are known, the density can be determined. This will be discussed briefly at the end of this subsection. Since the moments of a density are important, it would be useful if a function could be found that would give us a representation of all the moments. Such a function is called a moment generating function. Definition 20 Moment generating function Let X be a random variable with density fx(')' The expected value of etX is defined to be the moment generating function of X if the expected value exists for every value of t in some interval - h < t < h; h > O. The moment generating function, denoted by mx(t) or met), is
met)

= G[etX ] =
met)

foo etxfx(x) dx
-00

(19)

if the random variable X is continuous and is

= 8[etX ] =

L etxfx(x) x
IIII

if the random variable is discrete.

One might note that a moment generating function is defined in terms of a density function, and since density functions were defined without reference to random variables (see Definitions 6 and 9), a moment generating function can be discussed without reference to random variables. If a moment generating function exists, then met) is continuously differentiable in some neighborhood of the origin. If we differentiate the moment generating function r times with respect to t, we have
- , met) dt

d'

00

x'e x 1x(x)dx,

(20)

-00

and letting t -+ 0, we find

dt' m(O)

d'

8[X1 = Il;,

(21)

where the symbol on t~e left is to be interpreted to mean the rth derivative of met) evaluated as t -+ O. Thus the moments of a distribution may be obtained from the moment generating function by differentiation, hence its name. If in Eq. (19) we replace ext by its series expansion, we obtain the series expansion of met) in terms of the moments of fx( ); thus

'4

EXPECTATIONS AND MoMENrS

79

met)

4[1 +
II'

Xt

+ ~(Xt)2 + ~ (Xt)3 + ... ]

2! 3.

2 = 1 + r1 t + 2! r2 t + ...
II'

oo
j=

L0 I.

., rJ t ,
II.

1 ,

(22)

from which it is again evident that efficient of trlr!.

J.l; may be obtained from met); J.l; is the co-

EXAMPLE 17 Let X be a random variable with probability density function given by fx(X) = Ae- AX1[0, oo)(x).
mx(t) m'(t)

= 4[lX] = =
dm(t) dt

fo

oo

lXle-h dx

= --

1-t
m (0)

for t < 1.

1
(1 - t)2

hence

= 8[X] = -.
1
8[X2] =

And

" 21 m (t) = (1 _ t)3 '

so m"(O)

1~'

IIII

EXAMPLE 18 Consider the random variable X having probability density functionfx(x) = x- 2 /[1, oolx). (See Example 13.) If the momentgenerating function of X exists, then it is given by x- 2e tx dx. It can be shown, however, that the integral does not exist for any t > 0, and hence the moment generating function does not exist for this random variable X.

Jf

1II1
As with moments, there is also a generating function for factorial mODlents.

Definition 21 Factorial moment generating function Let X be a random variable. The factorial moment generating function is defined as 8[t X ] if this expectation exists. IIII
The factorial mODlent generating function is used to generate factorial moments in the same way as the raw moments are obtained from 8[e tX ] except that t approaches 1 instead of O. It sometimes simplifies finding moments of discrete distributions.

80

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

IJ

EXAMPLE 19

Suppose X has a discrete density function given by

e-;'A:~

fx(x) Then

= -

x!

for x = 0, 1, 2, ....

hence

d - B[ tX] dt

= A.
t=)

1111

In addition to raw moments, central moments, and factorial moments, there are other kinds of moments, called cumulants, or semi-invariants. Cumulants will be defined in terms of the cumulant generating function. We will not make use of cumulants in this book. ., Definition 22 Cumulant andcumulant generating function The logarithm of the moment generating function of X is defined to be the cumulant generating function of X. The rth cumulant of X, denoted by "r( X) or "r' is the coefficient of trlr! in. the Taylor series expansion of the cumulant generating function. 1111 A moment generating function is used, as its name suggests, to generate moments. That, however, will not be its only use (or us. An important use will be in determining distributions. Theorem 7 Let X and Y be two random variables with densities fx(') andfy(-), respectively. Suppose that mx(t) and my(t) both exist and are equal for all t in the interval - h < t < h for some h > O. Then the two cumulative distribution functions Fx (') and Fy(-) are equal. 111I A proof of the above theorem can be obtained using certain transfornl theory that is beyond the scope of this book. We should note, however, what the theorem asserts. It says that if we can find the moment generating function of a random variable, then, theoretically, we can find the distribution of the random variable since there is a unique distribution function for a given moment geneniting function. This theorem will prove to be extremely useful in finding the distribution of certain functions of random variables. In particular, see Sec. 4 of Chap. V.

PROBLEMS

81

EXAMPLE 20 'Suppose that a random yariab'le ,x has a moment generating function mx(t) = 1/(1 - t) for -}. <' t < 1; then we know that the density of X is given by /x(x) = e- x I[o,d:J) (x) since we showed in Example 17 above that Ae-.ax I[o,d:J)(x) has A/(A - t) for its moment generating function. / //1

Problem of moments We have seen that a density function determines a set of moments Ill, ~;, . .. when they exist. One of the im'portant problems in theoretical statistics is this: Given a set of moments, what is the density function from which these moments came, and is there only one density function that has these particular moments? We shall give only partial answers. First, there exists a sequence of moments for which there is an infinite (nondenumerable) collection of different distribution functions having these same moments. In general, a sequence of moments Il~, Il;, ... does not determine a unique distribution function. However, we did see that if the moment generating function of a random variable did exist, then this moment generating function did uniquely determine the corresponding distribution function. (See Theorem 7 above.) Hence, there are conditions (existence of'the moment generating function is a sufficient condition) under which a sequence of moments does uniquely determine a distribution function. The general problem of whether or not a distribution function is determined by its sequence of moments IS referred to as the problem 0/ moments and will not be discussed further.

PROBLEMS
J

(a)

Show that the following are probability density functions (p.d.f.'s):

hex) = e-x/(o.OO)(x} f2(X}
=

2e- 2X/(0. OO)(x}

f(x) =
(b)

() + l}};(x) -

()f2(X)
()2 =

Prove or disprove: If hex) and f2(x) are p.d.f.'s and if ()1 ()lh(x) + ()2f2(x) is a p.d.f. 2 Show that the following is a density function and find its median:
f(x)

1, then

a;2(a; + 2x) x(2ac + x) x2(a; +X)2 II.oo)(x)+ ac(ac + X)2

/(0.

, ](x), for a; > 0

3 Find the constant K so that the following is a p.d.f.

f(x)

= Kx 2 I(_K, KJ(X).

82

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

II

4 Suppose that the cumulative distribution function (c.d.f.) Fx(x) Can be written as a function of (x - a.)/fJ, where a. and fJ > 0 are constants; that is, x, a., and fJ appear in Fx( .) only in the indicated form. (a) Prove that if a. is increased by aa., then so is the mean of X. (b) Prove that if fJ is multiplied by k(k > 0), then so is the standard deviation of X. 5 The experiment is to toss two balls into four boxes in such a way that each ball is equally likely to fall in any box. Let X denote the number of balls in the first box. (a) What is the c.d.f. of X? (b) What is the density function of X? (c) Find the mean and variance of X. 6 A fair coin is tossed until a head appears. Let X denote the number of tosses required. (a) Find the density function of X. (b) Find the mean and variance of X. (c) Find the moment generating function (m.g.f.) of X. *7 A has two pennies; B has one. They match pennies until one of them has all three. Let X denote the number of trials required to end the game. (a) What is the density function of X? (b) Find the mean and variance of X. (c) What is the probability that B wins the game? 8 Let fx(x) =(1!fJ)[1 -I (x - a.)/fJl ]/(II-p. a+plx), where IX and (3 are fixed constants satisfying - 00 < a. < 00 and fJ > O. (a) Demonstrate that fx(') is a p.d.f., and sketch it. (b) Find the c.d.f. corresponding to fx(). (c) Find the mean and variance of X. (d) Find the qth quantile of X. 9 Letfx(x) = k(l/fJ){1 - [(x - a.)/{3]2}I(rJ-p. Cl+p,(X), where - 00 < IX < 00 and (3 > O. (a) Find k so that/xC') is a p.d.f., and sketch the p.d.f. (b) Find the mean, median, and variance of X. (c) Find 8[1 X - a.1] (d) Find the qth quantile of X. 10 Let fx(x) = t{O/(o. 1 ,(x) + 1[1. 2)(X) (1 - 0)/(2.3,(X)}, where 0 is a fixed constant satisfying 0 0 ~ 1. (a) Find the c.dJ. of X. (b) Find the mean, median, and variance of X. J1 Let f(x; 0) =' Of(X; 1) + (1 - O)f(x; 0), where 0 is a fixed constant satisfying o < 0 ~ 1. Assume that/(; 0) andf(; 1) are both p.d.f.'s. (a) Show that f( . ; 0) is also a p.d.f. (b) Find the mean and variance of f( ; 0) in terms of the mean and variance of f(' ; 0) and f(' ; 1), respectively. (c) Find the m.g.f. of/('; 0) in terms of the m.g.f.'s of/('; 0) andf(; 1).

PROBLEMS

83

J2 A bombing plane flies directly above a railroad track. Assume that if a large (small) bomb falls within 40 (15) feet of the track, the track will be sufficiently damaged so that traffic will be disrupted. Let X denote the perpendicular distance from the track that a bomb falls. Assume that
/x(x) (a) (b)

100-x l[o.loo)(x). 5000

Find the probability that a large bomb will disrupt traffic. If the plane can carry three large (eight small) bombs and uses all three (eight), what is the probability that traffic will be disrupted? 13 (a) Let X be a random variable with mean f' and variance a 2 Show that cf[(X - b)2], as a function of b, is minimized when b = 1-'. *(b) Let X be a continuous random variable with median m. Minimize cf[ I X b I] as a function of b. HINT: Show that 8[ IX - b I] = 8[ IX - m I] +
2
14

S; (x -

b}fx(x) dx.

(a)
(b)

If X is a random variable such that 8[X]

8[X2] = 13, use the ChebYShev inequality to determine a lower bound for P[ - 2 < X 8]. Let X be a discrete random variable with density /x(x) =l/(_l)(x)

= 3 and

+ f/(o)(x) + i/{1}(x).

(c)

For k = 2 evaluate P[I X -I-'xl > kax]. (This shows that in general the Chebyshev inequality cannot be improved.) If X is a random variable with 8[X] = I-' satisfying P[X ~ 0] = 0, show that

15

P[X> 21-'] ~ l. Let X be a random variable with p.d.f. given by

/x(x)

11

x I1[0. 2J(X).

Find the mean and variance of X. 16 Let X be a random variable having c.d.f.
Fx(x)

= pH(x)

(1 - p)G(x),

where p is a fixed real number satisfying 0 < p < 1,

and Sketch Fx(x) for p = i. (b) Give a formula for the p.d.f. of X or the discrete density function of X, whichever is appropriate. . (c) Evaluate p[X ~.I X ~ 1]. 17 Does there exist a random variable X for which P[p.x - 2ax X s;, I-'x + 2a x] = .6?
(a)

84

RANDOM VARIABLES, DISTRIBUTION FUNCTIONS, AND EXPECTATION

II

18

An urn contains balls numbered 1, 2, 3, First a ball is drawn from the urn, and then a fair coin is tossed the number of times as the number shown on the drawn ball. Find the expected number of heads. 19 If X has distribution given by P[X = 0] = P[X = 2] = p and P[X = 1] = 1 - 2p for 0 <p< i, for what p is the variance of X a maximum? 20 If X is a random variable for which P[X 0] = 0 and S[X] fL < 00, prove that P[X fLt] > 1 -l/t for every t 1. 21 Given the c.d.f.
Fx{x) = 0

for x < 0 forO <x <.5 for.5 for I

x< I

=
(a) (b)

<X.

Express Fx{x) in terms of indicator functions. Express Fx{x) in the form

aPC(x)

+ bP(x),

22

23 24

25

where Fac(.) is an absolutely continuous c.d.f. and P{) is a discrete c.dJ. (c) Find P[.25 < X .75]. (d) Find P[,25 < X < .5], Let/{x) = Ke-lIx(l - e-U)/(o. ro){x). (a) Find K such that /( .) is a density function. (b) Find the corresponding c.d.f. (c) Find P[X > 1], A coin is tossed four times. Let X denote the number of times a head is followed immediately by a tail. Find the distribution, mean, and variance of X. Let /x(x; () = ({)x + 1)/(- J. u(x), where () is a constant. (a) For what range of values of () is /x{ ; () a density function? (b) Find the mean and median of X. (c) For what values of () is var [X] maximized? Let X be a discrete random variable with the nonnegative integers as values. Note that S[tX] = of X, inasmuch as the coefficient of t J gives P[X = n. variable of Probs. 6 and 7.

L tJp[X = J=O

j].

Hence,S[t X] is a probability generating function

Find S[t X] for the random

III
SPECIAL PARAMETRIC FAMILIES OF UNIV ARIA TE DISTRIBUTIONS

1 INTRODUCTION AND SUMMARY

The purpose of this chapter is to present certain parametric families of univariate density functions that have standard names. A parametric family of density functions is a collection of density functions that is indexed by a quantity called a parameter. For example, letf(x; l) le-Ilx/(o. ro)(x), where A. > 0; then for each l > 0, f( ; l) is a probability density function. l is the parameter, and as A. ranges over the positive numbers, the collection {f( ; l): A. > O} is a parametric family of density functions. The chapter consists of three main sections: parametric families of dis.. crete densities are given in one; parametric families of probability density functions are given in another, and comments relating the two are given in the final section. For most of the families of distributions introduced, the means, variances, and moment generating functions are presented; also, a sketch of several representative members of a presented family is often included. A table summarizing results of Sees. 2 and 3 is given in Appendix B.

86

SPECIAL PARAMETRIC FAMIUES OF UNIVARIATE DlSTRIBUnONS

III

DISCRETE DISTRIBUTIONS

In this section we list several parametric families of univariate discrete densities. Sketches of most are given; the mean and variance of each are derived, and usually examples of random experiments for which the defined parametric family might provide a realistic model are included. The parameter (or parameters) indexes the family of densities. For each family of densities that is presented, the values that the parameter can assume will be specified. There is no uniform notation for parameters; both Greek and Latin letters are used to designate them.

2.1 Discrete Uniform Distribution

Definition 1 Discrete uniform distribution of discrete density functions 1
f(x) =f(x; N) =
N

Each member of the family

for x = 1, 2, ... , N otherwise

- I {l.Z . N} (x) , -N

(1)

where the parameter N ranges over the positive integers, is defined to have a discrete uniform distribution. A random variable X having a density 1111 given in Eq. (1) is called a discrete uniform random variable.

l/Nlu~
o
2 3

---~

__ _

FIGURE 1 Density of discrete uniform.

Theorem 1 (N + 1)/2,

If X has a discrete uniform distribution, then 8[X] =

(N Z -1) N 't 1 tX var [X] = 12 ' and mx(t) = 8[e ] = j~l e' N'

DISCRETE DISTRIBUTIONS

87

PROOF

var [X]

= S[X2] N(N

(S[X])2

= jf./2 N N

(N 2 + 1)2
(N

+ 1)(2N + 1)
6N

(N

+ 1)2
4

+ l)(N
]2

- 1)

11II
Remark The discrete uniform distribution is sometimes defined in density form as I(x; N) = [1/(N + l)]I{o. I. ... N} (x), for N a nonnegative integer. Jf such is the case, the formulas for the mean and variance have to be modified accordingly. /11I

2.2 Bernoulli and Binomial Distributions Definition 2 Bernoulli distribution A random variable X is defined to have a Bernoulli distribution if the discrete density function of X is given by

for x

0 or I}

= pX(l
otherwise where the parameter p satisfies 0 <p < 1.

- p)1 -xI lO l}(x),

(2)

p is often denoted by q.

111/

FIGURE 2

Bemou1li density.

88

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

Theorem 2 If X has a Bernoulli distribution, then

8[X] = p,
PROOF

var [X] = pq,

and

mx(t)

= pet + q.

(3)

8[X] = 0 . q + 1 . p = p. var [X] = 8[X2] - (8[X])2

mx(t) = 8[e tX ]

= 0 2 q + 12 p

_ p2

pq.

= q + pet.

IIII

EXAMPLE 1 A random experiment whose outcomes have been classified into two categories, called" success" and" failure," represented by the letters d and I, respectively, is called a Bernoulli trial. If a random variable X is defined as 1 if a Bernoulli trial results in success and 0 if the same Bernoulli trial results in failure, then X has a Bernoulli distribu1III tion with parameter p = P[success]. EXAMPLE 2 For a given arbitrary probability space (0, d, P[]) and for A belonging to d, define the random variable X to be the indicator function of A; that is, X(w) 1.4.(w); then X has a Bernoulli distribution with parameter p = P[X 1] = P[A]. II1I

Definition 3 Binomial distribution A random variable X is defined to have a binomial distribution if the discrete density function of X is given by

n) fx(x) =fx(x; n, p) = {( ~ p q
11

x n-x

for x = 0, I, ... , n otherwise

(4)

1 2

-l~.~, Illll
4 5 6 7 8 9 10

10. p =.25

11 ""

lO,p

.5

I o I 2

3 4

5 6

7 8 9 10

I
11

.. x

11 '"

5, P

.2

11 '"

5. P =.4

-= 5. p

,6

012345

FIGURE 3 Binomial densities.

DISCRETE DISTRIBUTIONS

89

where the two parameters nand p satisfy 0 <p ~ 1, n ranges over the positive integers, and q = I - p. A distribution defined by the density IIII function given in Eq. (4) is called a binomial distribution.

Theorem 3 If X has a binomial distribution, then

8[X]

= np,

var [X]

npq,

and

(5)

PROOF

mx(t) = 8[etX ] =

x=o

etx(n)pxqn-x =
X

x~o

(n)(petyqn-x
X

= (pet + qt.
Now

and

hence
S[X]

= m~(O)

= np

and var [X]

=

8[X2] - (8[X])2 mi(O) - (np)Z

= n(n - l)p2 + np - (np)2

= np(l - p).

I1II

Remark The binomial distribution reduces to the Bernoulli distribution when n = I. Sometimes the Bernoulli distribution is called the point
binomial.

IIII

EXAMPLE 3 Consider a random experiment consisting of n repeated independent Bernoulli trials when p is the probability of success a at each individual trial. The term "repeated" is used to indicate that the probability of 6 remains the same from trial to trial. The sample space for such a random experiment can be represented as follows:
Q
Zt

= {(ZI' zz, ... , zn): Zi = 6 or Zi =/}.

indicates the result of the "ith trial. Since the trials are independent, the probability of any specified outcome, say {(/' /' a, /, a, 6, ... , / , a)},

90

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

is given by qqpqpp ... qp. Let the random variable X represent the number of successes in the n repeated independent Bernoulli trials. Now P[X = x] = P[exactly x successes and n - x failures in n trials]
(:) p'q"- x for x
=

0, 1, ... , n since each outcome of the experimentthat has

exactly x successes has probability p"q'-x and there are (:) such outcomes.

Hence X has a binomial distribution.

I1I1

EXAMPLE 4 Consider sampling with replacement from an urn containing M balls, K of which are defective. Let X represent the number of defective balls in a sample of size n. The individual draws are Bernoulli trials where "defective" corresponds to "' success," and the experiment of taking a sample of size n with replacement consists of n repeated independent Bernoulli trials where p P[success] = KIM; so X has the binomial distribution for
x

= 0, 1, ... , n,

(6)

which is the same as P[Ad in Eq. (3) of Subsec. 3.5 of Chap. I, for x

= k.
1III

The sketches in Fig. 3 seem to indicate that the termsfx{x; n, p) increase monotonically and then decrease monotonically. The following theorem states that such is indeed the case.

Theorem 4 Let X have a binomial distribution with density fx(x; n, p); then fX<x - 1; n, p) <fx(x; n, p) for x < (n + l)p; fx(x - 1; n, p) > fx(x; n, p) for x > (n + l)p, andfx<x - 1; n,p) = fx(x; n, p) if x = (n + l)p and (n + l)p is an integer, where x ranges over 1, ... , n.
PROOF

ix(x; n,p) n- x fx(x - 1; n,p) x

+1

p
'-

1+

(n

+ 1)p xq

, + l)p, I111

which is greater than 1 if x < (n + l)p, smaller than 1 if x > (n and equal to 1 if the integer x should equal (n + l)p.

DISCRETE DISTRIBUTIONS

91

2.3

Hypergeometric Distribution Definition 4 Hypergeometric distribution A random variable X is defined to have a hypergeometric distribution if the discrete density function of X is given by

for
fx(x; M, K, n) =

x = 0, 1, ... , n

otherwise

(7)

Iro.I, .... n}(x)

where M is a positive integer, K is a nonnegative integer that is at most M, and n is a positive integer that is at most M. Any distribution function defined by the density function given in Eq. (7) above is called a hypergeometric distribution.

III1

Theorem 5

If X is a hypergeometric distribution, then

&[X] = n ' M
PROOF

and

var [Xl

= n.- .
M

K M-K . M-n M M-t

(8)

K)(M - K) ( 8[ Xl = t x x n- x =n. K
FO

(~)

(K - l)(M - K)
x-I

M x=,

(~~n

n - x

Ky - 1) (M n-l-y - 1- K + 1) ( =n .- L M (M - 1)
Kn-I

...:.----....;..---:~~-:-----.:;...~

y=O

n-l

- n 'M' using
i=O

f(a)( m b )=(a+b) - i m
i

given in Appendix A.

92

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

]0; K = 4; n = 4

= ]0; K

4; n

=5

__~__~~_~I__~.____. x
2 3 4

FIGURE 4 Hypergeometric densities.

8[X(X - 1)]

= Ix(x-l)
.=0

(~)(~=~)
(~)

=n(n_1)K(K-1) M(M - l)x=2

2) n-2 K- 2) (M - 2- K+ 2) ( n(n _ 1) K(K - 1) n- 2- Y = n(n _ 1) K(K - 1) . I M(M - 1) y""o (M - 2) M(M -1)

(M n

K- 2)(M - K) ( x-2 n-x

2

n-2
Hence var [X]

= 8[X2] = n(n

(8[X])2

= 6"[X(X +n

1)]

+ S[X] -

(6"[X])2

_ 1) K(K - 1) M(M-1)

K _ n 2 K2 M M2

K [ K-1 =n- (n-l). +1 M M-l

~]
1111

= nK

[(M - K)(M - n)]. M M(M -1)

Remark If we set KIM = p, then the mean of the hypergeometric distribution coincides with the mean of the binomial distribution, and the variance of the hypergeometric distribution is (M - n)/(M - 1) times the variance of the binomial distribution. 1111

DISCRETE DISTRIBUTIONS

93

EXAMPLE 5 Let X denote the number of defectives in a sample of size n when sampling is done without replacement from an urn containing M balls, K of which are defective. Then X has a hypergeometric distribution. IIII See Eq. (5) of Subsec. 3.5 in Chap. I.

2.4

Poisson Distribution Definition S Poisson distribution A random variable X is defined to have a Poisson distribution if the density of X is given by

fx(x)

= fx(x; A) =~

I
I

e-A.li X

x!

for

= 0, 1, 2, ... 1 e -A.'X ), r= 1 xl I (D. I .... }(x),

(9)

(0

otherwise The density given in Eq. (9) is

where the parameter Ii satisfies ), > O. called a Poisson density.

IIII

Theorem 6

Let X be a Poisson distributed random variable; then var [X] = Ii,

(10)

6"[X] = Ii,

.607
.303

.1.=t
.002 4

.368

.368 .184

.0I3 .~--~ 0 I 2 3

G
.073

-e--... x

I_J

A.-]

I
2

.06] 3

.015

.003 5

..x

_~.0!8

t
I

FIGURE 5 Poisson densities.

.005 IO

.002

---4---+ X

11

.001 12

94

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

PROOF

hence, and So,

G[X] = m~(O)
=

and

The Poisson distribution provides a realistic model for many random phenomena. Since the values of a Poisson random variable are the nonnegative integers, any random phenomenon for which a count of some sort is of interest is a candidate for modeling by assuming a Poisson distribution. Such a count might be the number of fAtal traffic accidents per week in a given state, the number of radioactive particle emissions per unit of time, the number of telephone calls per hour coming into the switchboard of a large business, the number of meteorites that collide with a test satellite during a single orbit, the number of organisms per unit volume of some fluid, the number of defects per unit of some material, the number of flaws per unit length of some wire, etc. Naturally, not all counts can be realistically modeled with a Poisson distribution, but some can; in fact, ifcertain assumptions regarding the phenomenon under observation are satisfied, the Poisson model is the correct model. Let uS assume now that we are observing the occurrence of certain happenings in time, space, region, or length. A happening might be a fatal traffic accident, a particle emission, the arrival of a telephone call, a meteorite collision, a defect in an area of material, a flaw in a length of wire, etc. We will talk as though the happenings are occurring in time; although happenings occurring in space or length are appropriate as well. The occurrences of the happening in time could be sketched as in Fig. 6. An occurrence of a happening is represented by x; the sketch indicates that seven happenings occurred between time 0 and ti.me t I' Assume now that there exists a positive quantity, say v, which satisfies the following:

DISCRETE DISTRIBUTIONS

95

~-~~----------~--~~-------*--*-~--------~--------x

FIGURE 6

(i) The probability that exactly one happening will occur in a small time interval of length h is approximately equal to vh, or prone happening in interval of length h] = vh + o(h). (ii) The probability of more than one happening in a small time interval of length h is negligible when compared to the probability of just one happening in the same time interval, or P[two or more happenings in interval of length h] = o(h). (iii) The num bers of happenings in nonoverlapping time intervals are independent. The term o(h), which is read some function of smaller order than 17," denotes an unspecified function which satisfies
H

lim o(h) h ..... O h

= 0.

The quantity v can be interpreted as the mean rate at which happenings occur per unit of time and is consequently referred to as the mean rate of occurrence. If the above three assumptions are satisfied, the number of occurrenceS of a happening in a period of time of length t has a Poisson distribution with parameter ). = vt. Or if the random variable Z(t) denotes the number of occurrenceS of the happening in a time interval of length t, then P[Z(t) = z] = e vt(vt)z/z! for z = 0, 1,2, .... We will outline two different proofs, neither of which is mathematically rigorous. For convenience, let t be a poi nt in time after time 0; so the time interval (0, t] has length t, and the time interval (t, t h] has length h. Let Pn(s) = P[Z(s) = n] = P[exactly n happenings In an interval of length s]; then
PROOF

Theorem 7

Poet

+ h)
=

=P[no happenings in interval (0, t

+ h]]
+ h]]

= P[no happenings in (0, t] andno happenings in (t, til]]

P[no happenings in (0, t]]P[no happenings in (t, t

= Po(t)Po(h),
using (iii), the independence assumption.

96

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

Now P[no happenings in (t, t + h]] = 1 - prone or more happenings in (t, t + h]] = 1 - prone happening in (t. t + h]] - P[more than one happening in (t, t + h]] = I - vh - o(h) - o(h); so Po(t + h) = Po(t) [1 - vh - o(h) - o(h)], or
poet

+ h) h

poet)

= -

"~poet) - Po t

( ) o(h)

+ o(h)

'

and on passing to the limit one obtains the differential equation P~Ct) = - vPo(t), whose solution is Po(t) = e vr, using the condition PoCO) = 1. Similarly, PtCt + h) = Pt(t)Po(h) + Po(t)Pt(h), or P1(t + h) = P.(t)[l - vh - o(h)] + PoCt)[vh + o(h)], which gives the differential equation P~(t) = - vPI(t) + vPo(t), the solution of which is given by PICt) = vte- vr , using the initial condition PI (0) = 0. Continuing in a similar fashion one obtains P~(t) = - vPII (t) + vP1I-.(t), for n = 2, 3, .... It is seen that this system of differential equations is satisfied by P II (t) = (vt)lI e -vt/n L The second proof can be had by dividing the interval (0, t) into, say n time subintervals, each of length h = tin. The probability that k happenings occur in the interval (0, t) is approximately equal to the probability that exactly one happening has occurred in each of k of the 11 subintervals that we divided the interval (0, t) into. Now the probability of a happening, or success," in a given subinterval is vh. Each subinterval provides us with a Bernoulli trial; either the subinterval has a happening, or it does not. Also, in view of the assumptions made, these Bernoulli trials are independent, repeated Bernoulli trials; hence the probability of exactly k "successes" in the n trials is given by (see Example 3)
H

(Z)( vh)k(l -

vh)"-t =

(~) [~n 1 - ~rk,

which is an approximation to the desired probability that k happenings will occur in time interval (0, t). An exact expression can be obtained by letting the number of subintervals increase to infinity, that is, by letting n tend to infinity;

n) [vt]k [1 _ W]II-k = _1 (vtt [1 _ _ vt]lI-k C_n)k -+ _( v_tt_e-_ (k n n k! n n k!

k

vt

as n -+

00,

. that [ ] 11 -+ e- vr, [ ] noting 1v -t -;; 1v -t -;;

-k

-+

1, and (n)klnk

-+

1.

//1/

DISCRETE DISTRIBUTIONS

97

Theorem 7 gives conditions under which certain random experiments involving counts of happenings in time (or length, space, area, volume, etc,) can be realistically modeled by assuming a Poisson distribution. The parameter v in the Poisson distribution is usually unknown. Techniques for estimating parameters such as v will be presented in Chap. VI I. . In practice great care has to be taken to avoid erroneously applying the Poisson distribution to counts. For example, in studying the distribution of insect larvae over some crop area, the Poisson model is apt to be invalid since insects lay eggs in clusters entailing that larvae are likely to be found in clusters, which is inconsistent with the assumption of independence of counts in small adjacent subareas.

EXAMPLE 6 Suppose that the average number of telephone calls arriving at the switchboard of a small corporation is 30 calls per hour. (i) What is the probability that no calls will arrive in a 3-minute period? Oi) What is the probability that more than five calls will arrive in a 5minute interval? Assume that the number of calls arriving during any time period has a Poisson distribution. Assume that time is measured in minutes; then 30 calls per hour is equivalent to .5 calls per minute, so the mean rate of occurrence is .5 per minute. P[no calls in 3-minute period] = e - vt = e-(S,(3) = e-I.s ~ .223. P[more than five calls in 5-minute interval]

= "

'Xl

e- vt( vt)k

k-6

'=-

k'
IIII

=2: k=6

00

(.S)(S)(2.5)k k! ~ .042.

EXAMPLE 7 A merchant knows that the number of a certain kind of item that he can sen in a given period of time is Poisson distributed. How many such items should the merchant stock so that the probability will be .95 that he will have enough items to meet the customer demand for a time period of length T? Let v denote the mean rate of occurrence per unit time and K the unknown number of items that the merchant should stock. Let X denote the number of demands for this kind of item during the time period of length T. The solution requires finding K so that P[X K]

> .95 or finding K so th.at

k=O

L [e-VT(vT)klk!] > .95.

In particular, if the

merchant sells an average of two such items per day, how many should

98

SPECIAL PARAMETRIC FAMIUES OF UNIVARIATE DISTRIBUTIONS

III

he stock so that he will have probability at least .95 of having enough items to meet demand for a 30-day month? Find K so that
K

e-(2)(30)60k

k~O
or find K so that
00

k!

> .95,

k==K+ )

The desired K can be found using an appropriate Poisson table (e.g., Molina, 1942 [45]). It is K = 73. 1111

EXAMPLE 8 Suppose that flaws in plywood occur at random with an average of one flaw per 50 square feet. What is the probability that a 4 foot x 8 foot sheet will have no flaws? At most one flaw? To get a solution assume that the number of flaws per unit area is Poisson distributed. P[no flaws] =
e- lo32 = e- 64

~ .527.
~ .865.

P[at most one flaw] = e- 64

+ .64e- 64

1111

A Poisson density function, like the binomial density, possesses a certain monotonicity that is precisely stated in the following theorem.

Theorem 8

Consider the Poisson density

e-A)..k

k!
(k-l)!

for k = 0, 1, 2, .... for k < ).., for k > )..,

e -A~k-I A e -;',k 1'. ---<-k!

e -A'k-I .I. e A1k A (k - I)! > -k-!-

and
I

e -A1k-) A

(k - I)!
PROOF

--~--~

k!

if ).. is an integer and k = )...

e-1).k-1/(k -l)! k e- A)..klk! ).. ,

which is less than 1 if k < l, greater than I if k > )., and equal to 1 if ).. is an integer and k = ).. 1111

DISCRETE DISTRIBUTIONS

99

2.5

Geometric and Negative Binomial Distributions

Two other families of discrete distributions that play important roles in statistics are the geometric (or Pascal) and negative binomial distributions. The reason that we consider the two together is twofold; first, the geometric distribution is a special caSe of the negative binomial distribution, and, second, the sum of independent and identically distributed geometric random variables is negative binomially distributed, as we shall see in Chap. V. In Subsec. 3.3 of this chapter, the exponential and gamma distributions are defined. We shall See that in several respects the geometric and negative binomial distributions are discrete analogs of the exponential and gamma distributions.

Definition 6 Geometric distribution A random variable X is defined

to have geometric (or Pascal) distribution if the density of X is given by

/x(x) =/x(x; p)
=

{:(l -

p)'

for x = 0, 1, ... }
= p(l - p)XI{o,
I, ... }(X),

(11)

otherwise where the parameter p satisfies 0 < p s,: I. (Define q = 1 - p.)

IIII

Definition 7 Negative binomial distribution A random variable X

with density
/x(X) = fx(x;r,p)

{~r +; - l)p'q' = (~r)v( -qt

( r

for

x = 0, 1, 2, ...
(12)

otherwise
=

+x

X-I) pq ,. xI

{O.I .... }X'

( )

where the parameters rand p satisfy r = I, 2, 3, ... and 0 < p < 1 (q = 1 - p), is defined to have a negative binomial distribution. The density given by Eq. (12) is called a negative binomial density.

IIII
~ema~k If i? the ne~a~ive binomial distribution r = 1, then the negative IIII bInOmIal c,tensity speCialIzes to the geometric density.

100

SPEOAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

-4

_l

Theorem 9 If the random variable X has a geometric distribution,

then
q C[X] =-, q var [X] = 2 ' P
p mx(t) = 1

and

-qe

(13)

Since a geometric distribution is a special case of a negative binomial distribution, Theorem 9 is a corollary of Theorem 11. IIII
PROOF

The geometric distribution is well named since the values that the geometric density assumes are the terms of a geometric series. Also the mode of the geometric density is necessarily o. A geometric density possesses one other interesting property, which is given in the following theorem.

Theorem 10 If X has the geometric density with parameter p, then

P[X > i

+ j] X > i] = P[X > j]

for i,j = 0, 1,2, ....

DISCRETE DISTRIBUTIONS

101

PROOF

P[X >

.,
I

+ JI X >

.
I]

P[X 2 i + j] P[X 2 i]

_ ;:.:...x=_.;....+...:;.l_--- _

00

p(1 -

PY

(l _p)i+l

~p(1
x=i

PY

(1-

= (1 - p)l
=P[X> j].

IIII

Theorem 10 says that the probability that a geometric random variable is greater than or equal to i + j given that it is greater than or equal to i is equal to the unconditional probability that it will be greater than or equal to j. We will comment on this again in the following example.

EXAMPLE 9 Consider a sequence of independent, repeated Bernoulli trials with p equal to the probability of success on an individual trial. Let the random variable X represent the number of trials required before the first succesS; then X has the geometric density given by Eq. (11). To See this, note that the first success will occur on trial x + 1 if this (x + I)st trial results in a succeSs and the first x trials resulted in failures; but, by independence, x successive failures followed by a SUCCeSS has probability (1 - PYp In the language of this example, Theorem 10 states that the probability that at least i + j trials are required before the first succeSS, given that there have been i successive failures, is equal to the unconditional probability that at least j trials are needed before the first success. That is, the fact that one has already observed i successive failures does not change the distribution of the number of trials required to obtain the first success. //1 I

A random variable X that has a geometric distribution is often referred to as a discrete waiting-time random variable. It represents how long (in terms of the number of failures) one has to wait for a SUCceSs. Before leaving the geometric distribution, we note that some authors define the geometric distribution by assuming 1 (instead of 0) is the smallest mass point. The density then has the form
!(x;p) = p(1 - pY- 1 I{l.2, ... }(x),

(14)

102

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

and the mean is lip, the variance isqlp2, and the moment generating function is petl( 1 - qe r ).

Theorem 11
S[X] =
rq

Let X have a negative binomial distribution; then var [X]

= 2

rq

'

and

m xC t)

=[

1 - qe

t] "',

(15)

PROOF

x.:O

f (- r)pr( -qe'Y = [ 1 - pqer]

X

[see Eq. (33) in Appendix A],

m~(t)

= pr( -r)(l

qe t ) r-I( _qe t )

and

hence
8[X] =
m~(t)
r=O

rq
p

and

var[X] = mit t)

0 -

(tf[X)2

= rqp'[qp -,-2(r + 1) + p-'-'] rq2 rq rq =-+-=-

(;r
/11/

p2

p2'

The negative binomial distribution, like the Poisson, has the nonnegative integers for its mass points; hence, the negative binomial distribution is potentially a model for a random experiment where a count of some sort is of interest, [ndeed, the negative binomial distribution has been applied in population counts, in health and accident statistics, in communications, and in other counts as welL Unlike the Poisson distribution, where the mean and variance are the same, the variance of the negative binomial distribution is greater than its mean. We will see in Subsec. 4,3 of this chapter that the negative binomial distribution can be obtained as a contagious distribution from the Poisson distribution.

DISCRETE DISTRIBUTIONS

103

EXAMPLE 10 Consider a sequence of independent, repeated Bernoulli trials with p equal to the probability of success on an individual triaL Let the random variable X represent the number of failures prior to the rth success; then X has the negative binomial density given by Eq. (12), as the following argument shows: The last trial must result in a success, having probability p; among the first x + r - I trials there must be r - I succeSSeS and x failures, and the probability of this is

r-l (x+r-l)

r-I

q -

x~

(r+x-l) x

r-Iqx

which when multiplied by p gives the desired result.

///1

A random variable X having a negative binomial distribution is often referred to as a discrete waiting-time random variable. It represents how long (in terms of the number of failures) one waits for the rth success.

EXAMPLE II The negative binomial distribution is of importance in the consideration of inverse binomial sampling. Suppose a proportion p of individuals in a population possesses a certain characteristic. Jf individuals in the population are sampled until exactly r individuals with the certain characteristic are found, then the number of individuals in eXcess of r that are observed or sampled has a negative binomial dis1//1 tribution.

2.6

Other'Discrete Distributions

In the previous five subsections we presented Seven parametric families of univariate discrete density functions. Each is commonly known by the names given. There are many other families of discrete density functions. In fact, new families can be formed from the presented families by various proceSses. One such process is called truncation. We will illustrate this process by looking at the Poisson distribution truncated at O. SUppose, as is sometimes the case, that the zero count cannot be observed yet the Poisson distribution Seems a reasonable model. One might then distribute the mass ordinarily given to the mass point 0 proportionately among the other mass points obtaining the family of densities for x = 1,2, ... otherwise.
(16)

104

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

HI

A random variable having density given by Eq. (16) is called a Poisson random variable truncated at O. Another process for obtaining a new family of densities from a given family can also be illustrated with the Poisson distribution. Suppose that a random variable X, representing a count of some sort, has a Poisson distribu w tion. If the experimenter is stuck with a rather poor counter, one that cannot count beyond 2, the random variable that the experimenter actually observes has density given by

z
fez)

o
e -A

2
,-..1. I - e -A - Ae

The counter counts correctly values 0 and 1 of the random variable X; but if X takes on any value 2 or more, the counter counts 2. Such a random variable is often referred to as a censored random variable. The above two illustrations indicate how other families of discrete densities can be fonnulated from existing families. We close this section by giving two further, not so wellwknown, families of discrete densities.

Definition 8 Beta-binomial distribution The distribution with discrete

density function

f(x) = I(x; n,

0:,

fJ) =

n) r(o: + f3) rex + o:)r(n + fJ r(o:)r(f3)' r(n + + f3) (x

0:

x)

I{o.

I ..... n}(x)

(17)

where n is a nonnegative integer, 0: > 0, and fJ > 0, is defined as the beta binomial distribution. rem) is the well-known gamma function rem) = xm - I e- X dx for m > O. See Appendix A. The beta-binomial distribution has
w

Io

Mean =

net.
Cl+

f3

and

. no:fJ(n +Cl+ f3) varIance = (Cl + fJ)2(et. + f3 + 1) .

(18)

If et. = fJ = 1, then the beta-binomial distribution reduces to a discrete uniform distribution over the integers 0, 1, "', n. IIII
It has the same mass points as the binomial distribution.

CONTINUOUS DISTRIBUTIONS

105

Definition 9 Logarithmic distribution The distribution with discrete

density function for
!(x;p) =
x

= 1, 2, ...

q. I ( ) 1 {1,2, ... } x, (19) -x oge P

otherwise

where the parameters satisfy 0 < P < 1 and q = 1 - p is defined as the logarithmic distribution. IIII The name is justified if one recalls the power-series expansion of loge (1 ~. q). The logarithmic distribution has .----

M ean

=----

q -p loge P

and

varIance =

q(q
(

+ loge p)

- plogeP

)2 .

(20)

[t can be derived as a limiting distribution of negative binomial distributions that have been generalized to include r, any positive number (rather than just an integer), truncated at O. The limiting distribution is obtained by letting r approach O.

CONTINUOUS DISTRIBUTIONS

In this section several parametric families of univariate probability density functions are presented. Sketches of some are incl uded; the mean and variance (when they exist) of each are given.

3.1

Uniform or Rectangular Distribution

A very simple distribution for a continuous random variable is the uniform dis-" tribution. It is particularly useful in theoretical statistics because it is convenient to deal with mathematically.

Definition 10 Uniform distribution If the probability density function of a random variable X is given by
/x(x) = /x(x; a, b) =

1 l[a bJ(X), b-a '

(21)

106

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

b-a

b - - - - - -........ X

---+-----a

FIGURE 8 Uniform probability density.

where the parameters a and b satisfy - 00 < a < b < 00, then the random variable X is defined to be uniformly distributed over the interval [a, b], and the distribution given by Eq. (21) is called a uniform distribution.

IIII
Theorem 12 if X is uniformly distributed over [a, b], then
(b - a)2 var [X] = - - 12 '
PROOF

e"t _
and

~t

mx(t) = (b - a)t'

(22)

8[X] =

1 {x b _ a dx
b

b2 - a 2 2( b - a)

+b
2 .

=
b~

var [X] = .8'[X2] - (.8'[X])2 = {X2

adx _ (a ; b)'

b 3 -a 3 3(b-a)

(a+b)2 4

(b-a)2 12

IIII
The uniform distribution gets its name from the fact that its density is uniform, or constant, over the interval [a, b]. It is also called the rectangular distribution-the shape of the density is rectangular. The cumulative distribution function of a uniform random variable is given by (23)

CONTINUOUS DISTRIBUTIONS

107

It provides a useful model for a few random phenomena. For instance, if it is known that the values of some random variable X can only be in a finite interval, say [a, b], and if one assumes that any two subintervals of [a, b] of equal length have the same probability of containing X, then X has a uniform distribution over the interval [a, b]. When one speaks of a random number from the interval [0, 1], one is thinking of the value of a uniformly distributed random variable over the interval [0, 1]. EXAMPLE 12 If a wheel is spun and then allowed to come to rest, the point on the circumference of the wheel that is located opposite a certain fixed marker could be considered the value of a random variable X that is uniformly distributed over the circumference of the wheel. One could 11II then compute the probability that X will fall in any given arc. Although we defined the uniform distribution as being uniformly distributed over the closed interval [a, b], one could just as well define it over the open interval (a, b) [in which case/x(x) = (b - a) II(a,b)(x)] or over either of the ha/f-open-ha/f-c1osed intervals (a, b] or [a, b). Note that all four of the possible densities have the same cumulative distribution function. This lack of uniqueneSS of probability density functions was first mentioned in Subsec. 3.2 of Chap. II.

3.2

Normal Distribution

A great many of the techniques used in applied statistics are based upon the normal distribution; it will frequently appear in the remainder of this book.

Definition 11 Normal distribution A random variable X is defined to be normally distributed if its density is given by
/x(x)

= /x(x; p"

q) =

JI

e-(X-Jl,)2/2cr\

(24)

2nq

where the parameters p, and q satisfy - (X) < p, < 00 and q > 0. Any distribution defined by a density function given In Eq. (24) is called a normal distribution. IIII We have used the symbols p, and q2 to represent the parameters because these paramete~s t,urn .out, as we shan see, to be the mean and variance, respectively, of the distrIbution.

108

SPECIAL PARAMETRIC FAMIUES OF UNIVARIATE DISTRIBUTIONS

III

FIGURE 9 Norma] densities.

"

'

One can readily check that the mode of a normal density occurs at x = Jl and inflection points occur at Jl - a and Jl + a. (See Fig. 9.) Since the normal distribution occurs so frequently in later chapters, special notation is introduced for it. If random variable X is norma]]y distributed with mean J1 and variance a 2, we wi]] write X,.... N(J1, ( 2). We will also use the notation </J/1. a2(x) for the density of X,.... N(Jl, ( 2) and <1>/1. a2(x) for the cumulative distribution function. If the normal random variable has mean 0 and variance 1, it is called a standard or normalized normal random variable. For a standard normal random variable the subscripts of the density and distribution function notations are dropped; that is,
</J(x),= tx2 J~e2n

and

<1>(x) =

IX
-00

</J(u) duo

(25)

Since </J1l. a2(x) is given to be a density function, it is implied that

I_
00 00

</J /1 , a 2 (x) dx = 1,

but we should satisfy ourselves that this is true. The verification is somewhat troublesome because the indefinite integral of this particular density function does not have a simple functional expression. Suppose that we represent the area under the curve by A; then

A=
and on making the substitution y
=

1 foo e-(x-Il)2j2a2d x, J2na -00

(x - Jl)/a, we find that

1 -ty2 d A=J~ e y. 2n - 00

I oo

CONTINUOUS DISTRIBUTIONS

109

FIGURE 10 Norma] cumulative distribution function.

We wish to show that A = 1, and this is most easily done by showing that A2 is 1 and then reasoning that A = 1 since <P 1l ,a2 (x) is positive. We may put
A=J.~
2

foo
-00

2n

-ty2

foo e -tz Y 1J 2n
-00

= -1
2n

foo foo
-00 -00

e- t (y2+ z 2) dy dz

by writing the product of two integrals as a double integral. In this integral we change the variables to polar coordinates by the substitutions

y=
and the integra] becomes

r sin

z = r cos

e,

=1.

Theorem 13 If X is a normal random variable,

&[ X] = J.l,
PROOF

and

(26)

mx(t) = C[e tX ]
=
e tll

oo
1

= llltS'[l(X-Il)]
-= l(x- ll )e-(1/2a2)(x-Il)2 dx
J2n
1

-00

= etll

J2n

foo
-00

e-O/2(2)[(x-IlP-2a2t(x-ll)]dx.

I f we complete the square inside the bracket, it becomes

(x - /1)2 - 2(12t(X - J.l)

= (x = (x -

J.l)2 - 2CT 2t(x - /1) J.l - (12t)2 - (14 t 2,

+ (14 t 2 _

(14t 2

110

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

In

and we have
mx(t) = etJletl2t2/2

foo e-(x-Jl-a2t)2/2t12 dx.

-00

J21tu

The integra] together with the factor I IJ21tU is necessarily I since it is the area under a normal distribution with mean Jl + u 2 t and variance a 2 Hence,
mx(t) =
eJlt+a t /2,
2 2

On differentiating mx(t) twice and substituting t and var [X]

=

= 0,

we find

8[X] = m~(O) 8[X2] - (8[X])2 = mi.-(O) - Jl2

= jJ
a 2,

thus justifying our use of the symbols Jl and

u2 for the parameters. IIII

Since the indefinite integral of 4J Jl a2(x) does not have a simple functional form, one can only exhibit the cumulative distribution function as
<I>Jl,a2(x) =

f- oo 4Jp.,a2(u) duo
x

(27)

The folIowing theorem shows that we can find the probabiHty that a normally distributed random variable, with mean Jl and variance a 2 , falls in any interval in terms of the standard norma] cumulative distribution function, and this standard normal cumulative distribution function is tabled in Table 2 of Appendix O.

Theorem 14 If X '" N(p, a2 ), then

PIa
PROOF

< X <hJ

= fl>

e: 1') - e 1').
fl>

(28)

P[a

< X < bJ =

e- H (x- Jl )/a]2 dx

J21tU
1

f (b-Jl)/a
(a-Jl)/a

J21t e

-tz 2 d

CONTINUOUS DISTRIBUTIONS

111

Remark

<I>(x) = 1 - <1>( - x).

IIII

The normal distribution appears to be a reasonable model of the behavior of certain random phenomena. Jt also is the limiting form of many other prob~ ability distributions. Some such limits are given in Subsec. 4.1 of this chapter. The normal distribution is also the limiting distribution in the famous centra/limit theorem, which is discussed in Sec. 4 of Chap. V and again in Sec. 3 of Chap. VI. Most students are already somewhat familiar with the normal distribution because of their experience with "grading on the curve." This notion is covered in the fol1owing example.
EXAMPLE 13

Suppose that an instructor assumes that a student's final score is the value of a norma]]y distributed random variable. If the instructor decides to award a grade of A to those students whose score exceeds J1 + a, a B to those students whose score fa]]s between Jl and }J. + a, a C if a score falls between J1 - (j and Jl, a D if a score falls between J1 - 2a and J1 - a, and an F if the score falls below J1 - 2a, then the proportions of each grade given can be calculated. For example, since

PI X > J1 + 0"] =

1-

PI X < Jl + 0"] =
~

1 - <\l

(Jl + ; - Jl)
IIII

= 1 - <I>(l)

.1587,

one would expect 15.87 percent of the students to receive A's.

EXAMPLE 14

Suppose that the diameters of shafts manufactured by a certain machine are normal random variables with mean 10 centimeters and standard deviation .1 centimeter. If for a given application the shaft must meet the requirement that its diameter fa]] between 9.9 and 10.2 centimeters, what proportion of the shafts made by this machine wi]] meet the requirement?
P[9.9

< X<

10.2] = <\l Co.~ 1- 10) -

<\l

e9.~

10)
IIII

= <1>(2) - <1>(-1) ~ .9772 - .1587 = .8185.

3.3

Exponential and Gamma Distributions

Two other families of distributions that play important roles in statistics are the (negative) exponential and gamma distributions, which are defined in this subsection. The reason that the two are considered together is twofold; first, the

112

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

exponential is a special case of the gamma, and, second, the sum of independent identically distributed exponential random variables is gamma-distributed, as we shall see in Chap. V.

Definition 12 Exponential distribution If a random variable X has a density given by

/x(x; A) = Ae-AxI[o. oolx),
~.

(29)

where A > 0, then X is defined to have an (negative) exponential distribu-

Definition 13 Gamma distribution If a random variable X has density given by

where r > and A > 0, then X is defined to have a gamma distribution. r() is the gamma function and it is discussed in Appendix A. IIII

Remark If in the gamma density to the exponential density.

= 1, the gamma density specializes

/III

Theorem 15 If X has an exponential distribution, then

1 4[X] =-,

var [Xl

= A2 '

and

mx(t) = - -

A A- t

for
(31)

The exponentia.l distribution was the distribution used as an example for some definitions given in Chap. II, and derivations of the above appear there. Also, Theorem 15 is a corol1ary to the following theorem.
PROOF

fill
Theorem 16 If X has a gamma distribution with parameters r and A, then
var [X] = A2 '
r

and

mx(t)

). )r (
A-t

for t < A.
(32)

CONTINUOUS DISTRIBUTIONS

113

1.0

o~~~~~~~~~~~~7~~8~X 1 2
FIGURE 11 Gamma densities (A
I),

PROOF

=
m~(t)=

A (A - ty x (A - t 0 r(r) rArO. - t)-r-I

)r Joo

r-I e

-(A.-t)x

d x_ (_A . A- t

)r

and

mHt) = r(r
hence

+ l)Ar(A -

t)-r-2;

4'[X] =

m~(O) =

i
IIII

and var [X] = 4'[X2] - (~[X])2

= mi(O) _

(!:.)2 =
1

r(r

~ 1) _ (~) 2_ r
1

The exponential distribution has been used as a model for lifetimes of various things. When we introduced the Poisson distribution, we spoke of certain happenings, for example, particle emissions, occurring in time. The length of the time interval between successive happenings can be shown to have an exponential distribution provided that the number of happenings in a fixed

114

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

time interval has a Poisson distribution. We comment on this again in Subsec. 4.2 below. Also, if we assume again that the number of happenings in a fixed time interval is Poisson distributed, the length of time between time 0 and the instant when the rth happening occurs can be shown to have a gamma distribution. So a gamma random variable can be thought of as a continuous waitingtime random variable. It is the time one has to wait for the rth happening. Recall that the 'geometric and negative binomial random variables were discrete waiting-time random variables. In a sense, they are discrete analogs of the negative exponential and gamma distributions, respectively.

Theorem 17 If the random variable X has a gamma distribution with

parameters r and A, where r is a positive integer, then
r - 1

Fx(x) = 1 -

e - ).x ().x)j
J.

.,

(33)

j=O

PROOF

The proof can be obtained by succeSSIve integrations by

parts.

IIII

For A = 1, Fx(x) given in Eq. (33) is cal1ed the incomplete gamma/unction and has been extensively tabulated.

Theorem 18 If the random variable X has an exponential distribution

wi th parameter A, then
P[X> a

+ bl X>
P[X> a

a] =P[X> b],

for a > 0 and b > O.

P[X> a + b] P[X> a]
e-).(a+b) e-).a

PROOF

+ blX> a] =

= e-)'b = P[X >

b].

IIII

Let X represent the lifetime of a given component; then, in words, Theorem 18 states that the conditional probability that the component wi1I last a + b time units given that it has lasted a time units is the same as its initial probability of lasting b time units. Another way of saying this is to say that an "old" functioning component has the same 1ifetime distribution as a new" functioning component or that the component is not subject to fatigue or to wear.
H

CONTINUOUS DISTRIBUTIONS

115

3.4 Beta Distribution

A family of probability densities of continuous random variables taking on values in the interval (0, 1) is the family of beta distributions,

Definition 14 Beta distribution If a random variable X has a density given by

/x(x) = /x(x; a, b)

B(~, b) x"-t (I

- x)'-t 1(0. t)(x),

(34)

where a >

and b > 0, then X is defined to have a beta distribution.

n
X)b-l

IIII

The function B(a, b) = xa-1(1 mentioned briefly in Appendix A.

dx, caned the beta junction, is

Remark The beta distribution reduces to the uniform distribution over (0, 1) if a = b = 1. IIII Remark The cumulative distribution function of a beta-distributed random variable is
F x(x; a, b)

= 1(0, l)(X)

1
B(a, b) u

(l -

U)b-l

du

+ 1[1, ooix);

(35)

it is often called the incomplete beta and has been extensively tabulated.

IIII

2.0

I.S
~~~~

__+-______~~~~lb=l

a = 1

.5

FIGURE 12 Beta densities.

~~~----~

__

____

__

~L-_x

.4

.6

.8

116

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

The moment generating function for the beta distribution does not have a simple form; however the moments are readily found by using their definition.

Theorem 19 If X is a beta-distributed random variable, then

~[X] =
PROOF
~[Xk] =

a a+b

and

VM[X]=

ab
(a

+b+

l)(a

+ b)2

1 B(a, b)

II
0

y!+a-l(1 - X)b-l dx

B(k + a, b) B(a, b) _ r(k + a)r(a - r(a)r(k + a

hence,

r(k + a)r(b) . r(a + b) r(k + a + b) r(a)r(b)

+ b). + b)'

~[X] = r(a + l)r(a + b) = a r(a)r(a+b+l) a+b'

and var [X]

=

~[X ] - (~[X])
(a

r(a + 2)r(a r(a)r(a + b (

+ b) ( a ) 2 + 2) - a + b
IIII

a )2 ab =(a+b+l)(a+b)- a+b =(a+b+l)(a+b)2

+ 1)a

The family of beta densities is a two-parameter family of densities that is positive on the interval (0, 1) and can assume quite a variety of different shapes, and, consequently, the beta distribution can be used to model an experiment for which one of the shapes is appropriate.

3.5

Other Continuous Distributions

In this subsection other parametric families ofprobabiHty density functions that will appear later in this book are briefly introduced; many other families exist. The introductions of the three families of distributions, that go by the names of Student's t distribution, chi-square distribution, and F distribution, are deferred until Chap. VI. These three families, as we sha11 see, are very important when sampHng from normal distributions.

CONTINUOUS DISTRIBUTIONS

117

Cauchy distribution A distribution which we shaH find useful for illustrative

purposes is the Cauchy, which has the density
fx(x; ct, {J)

= n{J{1 +

1 [(x _ ct)/{J]2} '

(36)

where - 00 < ct < 00 and {J > O. Although the Cauchy density is symmetrical about the parameter ct, its mean and higher moments do not exist. The cumulative distribution function
IS

F (x) - -

1 - n
= -

IX
-00

du n{J{l + [(u - a)/{J]2}

(37)

X-ct - arc tan - n {J

Lognormal distribution

Let X be a positive random variable, and let a new random variable Y be defined as Y = loge X. If Y has a normal distribution, then X is said to have a lognormal distribu tion. The density of a lognormal distribu tion is given by

f
where
-00

(x, 11,

(1 ) -

2 _ J1 exp [1 - 2 x 2n(1 (1
and
(1

(Ioge x - 11)

2]

[(0.

OO)(x),

(38)

< 11 <

00

> O.
and

C[X]

= eJl+t

a2

(39)

for a lognormal random variable X. Also, if X has a lognormal distribution, then C[loge (X)] = 11, and var [1oge (X)] = (12.

Double exponential or Laplace distribution A random variable X is said to

have a double exponential, or Laplace, distribution if the density function of X is given by
fx(X) = fx(x; ct, {J) = _I exp (_ Ix - ct l) 2{J {J'

(40)

where -

00

< ct <

00

and {J > O.

If X has a Laplace distribution, then

C[X]

= ct

and

(41)

Weibull distribution The density

f(x; a, b) = ab:x!'-le-aXb[(O, ooix)

(42)

118

SPECIAL PARAMETRIC FAMIUES OF UNIVARIATE DISTRIBUTIONS

III

where a > 0 and b > 0, is ca]]ed the Weibull density, a distribution that has been successfully used in reliability theory. For b = 1, the Weibu11 density reduces to the exponential density. 1t has mean (1/a)llhr(l + b- 1 ) and variance (l/a)2/h[r(1 + 2b- 1 ) - r2(l + b- 1 )).

Logistic distribution The logistic distribution is given in cumulative distribution form by

F( x . r:x

"

fJ) ___1-,----.-:--: - 1 + e-(x-a)lfJ'

. (43)

where - 00 < r:x < 00 and fJ > O. The mean of the logistic distribution is given by r:x. The variance is given by fJ 2 n 2 /3. Note that F( r:x - d; r:x, fJ) = I - F(r:x + d; r:x, fJ), and so the density of the logistic is symmetrical about cx. This distribution has been used to model tolerance levels in bioassay problems.

Pareto distribution The Pareto distribution is given in density-function form

by l44) where () > 0 and Xo > O. distribution are given by The mean and variance respectively of the Pareto

()xo

()-I

for

() > 1

and

()x5 _ ( ()x o )
()-2 ()-I

for

() > 2.

This distribution has found application in modeling problems involving distributions of incom.es when incomes exceed a certain limit Xo .

Gumbel distribution The cumulative distribution function

F(x; cx, fJ) = exp( - e -(x-a)lfJ),

(45)
It appears

where - 00 < r:x < 00 and fJ> 0 is ca]]ed the Gumbel distribution. as a limiting distribu tion in the theory of extreme-value statistics.

Pearsonian system of distributions Consider a density function /x(x)

which satisfies the differential equation

d/x(x) /x(X) dx
---

+a 2 bo + b1x + b 2 x
x

(46)

for constants a, bo , b l , and b 2

Such a density is said to belong to the Pearsonian

COMMENTS

119

system of density functions. Many of the probability density functions that we have considered are special cases of the Pearsonian system. For example, if
I\,

~r

fxCx) =
then

r -1

er(r)

AX

I[o,oo)Cx),

dfxC x) fx(x) dx

--

= -A +

r - J

x - (r - 1)/A = -----

-x/).

for x > 0; so the gamma distribution is a member of the Pearsonian system with a = -(r - 1)/)" b I = -1/)', and bo = b 2 = O.

COMMENTS

We conclude this chapter by making severa) comments that tie together some of the density functions defined in Secs. 2 and 3 of this chapter.

4.1 . Approximations
Although many approximations of one distribution by another exist, we wi]] give only three here. Others wiJ] be given along with the central-limit theorem in Chaps. V and VI.

Binomial by Poisson We defined the binomial discrete density function,

with parameters nand p, as for x 0, 1, ... , n.

If the parameter n approaches infinity and p approaches 0 in such a way that np remains constant, say equal to )., then (47) for fixed integer x. sideration: The above fo11ows immediately from the fo]]owing con-

n)pX(1 _ p)n-x ( x.

= (n}x (~)X(l _~)n-x

xI
n n

120

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

since

A) (1-;;

-x

-+1,

and

as

n -+

00.

Thus, for large n and small p the binomial probability(:)r(l _ p)O-X

can be approximated by the Poisson probability e-np(npr/x!. The utility of this approximation is evident if one notes that the binomial probability involves two parameters and the Poisson only one.
Binomial and Poisson by normal Theorem 20 Let random variable X have a Poisson distribution with parameter A; then for fixed a < b
P[a < JA
= P[A

X-A

< b]
X < A + bJJ.] -+ CJ>(b) - <1>(a)

+ aJ~ <

as

A -+

00.

(48)

Omitted. [Eq. (48) can be proved using Stirling's formula, which is given in Appendix A. It also fol1ows from the central-limit 1/11 theorem.] I, /1
PROOF

Theorem 21 De Moivre-Laplace limit theorem Let a random variable X have a binomial distribution with parameters nand p; then for fixed
a<b

CJ>(b) - <1>(a)
PROOF

as n -+ 00.

(49)

Omitted. (This is a special case of the central-limit II1I theorem, given in Chaps. V and VI.)
Remark We approximated the binomial distribution with a Poisson distribution in Eq. (47) for large n and small p. Theorem 21 gIVes a normal approximation of the binomial distribution for large n. IIII

The usefulness of Theorems 20 and 21 rests in the approximations that they give. For instance, Eq. (49) states thatP[np + aJnpq < X <np + bJnpq]

COMMENTS

121

is approximately equal to <I>(b) - <I>(a) for large n. Or if c = np + aJnpq and d = np + bJnpq, then Eq. (49) gives that P[c < X < d] is approximately equal to
<I> ( Jnpq

d - np )

<I>

(C - np )
Jnpq

for large n, and, so, an approximate value for the probability that a binomial random variable faIJs in an interval can be obtained from the standard norma] distribution. Note that the binomial distribution is discrete and theapproximating norma] distribution is continuous.

EXAMPLE 15 Suppose that two fair dice are tossed 600 times. Let X denote the number of times a total of 7 occurs. Then X has a binomial distribution with parameters n = 600 and p = ~. 8[X] = 100. Find P[90 < X < 110].
P[90

< 11 0] = if (600) (~)j(~)600j=90 j 6 6 '

j

a sum that is tedious to evaluate. Eq. (49), we have

P[90

Using the approximation given by

<

< 110] ~ <I> ( J.i~O

110 - 100)

<I>

(90 - 100)

Jsgo

<I>(Ji) - <1>( -J~) ~ <1>(1.095) - <1>( -1.095) ~ .726.

IIII
4.2 Poisson and Exponential Relationship

When the Poisson distribution was introduced in Subsec. 2.4, an experiment consisting of the counting of the number of happenings of a certain phenomenon in time was given special consideration. We argued that under certain conditions the count of the nmriber of happenings in a fixed time interval was Poisson distributed with parameter, the mean, proportional to the length of the interval. Suppose now that one of these happenings has just occurred; what then is the distribution of the length of time, say X, that one will have to wait until the next happening? P[X> t] = P[no happenings in time interval of length t] = e- vt , where v is the mean occurrence rate; so
Fx(t)

= P[X ~ t]

= 1 - P[X >

t] = 1 _ e- vt

for t > 0;

122

SPECIAL PARAMETRIC FAMIUES OF UNIVARIATE DISTRIBUTIONS

III

that is, X has an exponential distribution. On the other hand, it can be proved, under an independence assumption, that if the happenings are occurring in time in such a way that the distribution of the lengths of time between successive happening~ is exponential, then the distribution of the number of happenings in a fixed time interval is Poisson distributed. Thus the exponentia1 and Poisson distribu tions are re1ated.

4.3

Contagious Distributions and Truncated Distributions

A brief introduction to the concept of contagious distributions is given here. If/o('), 11('), ... ,/,.(.), ... is a sequence of density functions which are either al1 discrete density functions or all probabiHty density functions which mayor may not depend on parameters, and Po, PI' ... , Pn , is a sequence of parameters satisfying Pi > 0 and
i=O

L Pi = 1, then L Pih(x) is a density

i=O

00

00

function, which is

sometimes called a contagious distribution Or a mixture. For example, if lo(x) = 4>"0. u02 (x) (a norma1 with mean Ilo and variance O'~) and 11 (x) = 4>lll. U1 2 (X), then

where PI = P and Po = 1 - p, is a mixture of two normal densities. Equation (50) is a1so sometimes referred to as a contaminated normal. A random variab1e Xhas distribution given by Eq. (50) ifit is normal1y distributed with mean III and variance O'i with probability P and normally distributed with mean Ilo and variance O'~ with probabiJity 1 - p. Contagious distributions or mixtures can be useful mode1s for certain experiments. For instance, the mixture of two normal distributions given in Eq. (50) has five parameters, name1y, p, Ilo, Ill' 0'0, and 0' l' If we vary these five parameters, the density can be forced to assume a variety of different shapes, some of which are bimoda1; that is, the density has two distinct 10cal maximums. Physical considerations of the random experiment at hand can sometimes persuade one to consider mode1ing the experiment with a mixture. The experimenter may know that the phenomena that he is observing are a mixture; for example, the radioactive partic1e emissions under observation might be a mixture of the partic1e emissions of two, or several, different types of radioactive materia1s.

COMMENTS

123

The concept of mixing can be extended. Let {f(x; 8)} be a family of density functions parameterized or indexed by 8. Let the totaHty of values that the parameter 8 can assume be denoted by e. If e is an interval (possibly infinite) and g(8) is a probability density function which is for an arguments not in e, then

J~f(x; 8)g(8) d8

(51)

is again a density function, ca]]ed a contagious distribution or a mixture. For example, supposef(x; 8) = e-08 x /x! for x = 0,1,2, ... andf(x; 8) = otherwise and

g(O)

r(r) u

A'

l l ' -1

;01

(O,(X)

(ll)
U ,

a gamma density.

Then

A'

x!r(r) A' . r(r

(X)

O,+x-1 e -O.+ 1)0 d8

+ x)

I(X)

[(A

+ 1)0j'+x-1 e -().+ 1)8 d[(A + 1)8]

r(r

x!r(r) (A + 1Y+x

+ x)

A )' r(r + x) 1 ( - A + 1 (x!)r(r) (A + It

(r +xx- 1) ().+1 A )'( 1 ) A+1

for

x = 0, 1, ... ,

which is the density function of a negative binomial distribution with parameters rand P =A/(A + 1). We say that the derived negative binomial distribution is the gamma mixture of Poissons.

is sometimes called a compound Poisson, where g(O)/(o,(X)iO) is a probability density function. We have sketchily i11ustrated above how new parametric families of densities can be obtained from existing families by the technique of mixing. In Subsec. 2.6 we indicated how truncation could be employed to generate new families of discrete densities. Truncation can also be utilized to form other families of continuous distributions. For instance, the family of beta distributions provides densities that are useful in modeling an experiment for which

124

SPECIAL pARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

it is known that the values that the random variable can assume are between 0 and 1. A truncated norma] or gamma distribution would also provide a useful model for such an experiment. A normal distribution that is truncated at 0 on the left and at 1 on the right is defined in density form as

ji() x = ji( x; /1, (1 )

<Pp.,a2(x)l(o,l)(x) ct> p.,a2( 1) - ct>ll,a2(O)

(52)

This truncated normal distribution, like the beta distribution, assumes values between 0 and 1. Truncation can be defined in general. If X is a random variable with density Ix(-) and cumulative distribution Fx('), then the density of X truncated on the left at a and on the right at b is given by Ix(x)l(a,b)(X) (53) Fx(b) - Fx(a) .

PROBLEMS
1 (a)
(b)
(c)

(d)
(e)

(/)
(g) (h)

(0
(j) (k)

(/)
(m) (n) (0) (p)

2 (a)
(b)

Let X be a random variable having a binomial distribution with parameters n 25 and p = .2. Evaluate P[X < ftx 2ax]. If X is a random variable with Poisson distribution satisfying P[X 0] = P[X = 1], what is G[X]? If X is uniformly distributed over (1, 2), find z such that P[X > z+ ftx] L If X is normally distributed with mean 2 and variance 1, find P[I X 21 < 1]. Suppose X is binomial1y distributed with parameters nand p; further suppose that G[X] = 5 and var [X] = 4. Find nand p. If G[X] = 10 and ax = 3, can X have a negative binomial distribution? If Xhas a negative exponential distribution with mean 2, find P[X < 11 X < 2]. Name three distributions fOr which P[X < ftx] = ~.. Let X be a random variable having binomial distribution with parameters n = 100 andp =.1. Evaluate P[X < ftx - 3ax]. If X has a Poisson distribution and P[X = 0] = i, what is G[X]? Suppose X has a binomial distribution with parameters n and p. For what p is var [X] maximized if we assumed n is fixed? Suppose X has a negative exponential distribution with parameter A. If P[X 1] = P[X > 1], what is var [X]? Suppose X is a continuous random variable with uniform distribution having mean 1 and variance t. What is p[X < OJ? If X has a beta distribution, can G[lt X] be unity? Can X ever have the same distribution as - X? If so, when? If X is a random variable having moment generating function exp (e 1), what is 8[X]? Find the mode of the beta distribution. Find the mode of the gamma distribution.
l -

PROBLEMS

125

3 'Name a parametric family of distributions which satisfies: (a) The mean must be greater than or equal to the variance. (b) The mean must be equal. to the variance. (c) The mean must be less than or equal to the variance. (d) The mean can be less than, equal to, or greater than the variance (for different parameter values). 4 (a) If X is norma]]y distributed with mean 2 and variance 2, express P[ I X - i I <2] in terms of the standard normal cumulative distribution function. (b) If X is norma]]y distributed with mean ft > 0 and variance (12 = ft\ express P[X < - ft I X < ft] in terms of the standard normal cumulative distribution function. (c) Let X be normally distributed with mean ft and variance (12. Suppose (12 is some function of ft, say (12 = h(ft). Pick h(') sO that P[X < 0] does not depend on J1 for ft > O. 5 Use the ahernate definition of the median as given in the remark fo]]owing Definition 18 of Chap. II. Find the median in each of the fo]]owing cases:
(a) /x(x)

;"e-;'X[(O,

a(x),

is uniformly distributed on the interval (0 1, (J2). has a binomial distribution with n = 4, p = .5. has a binomial distribution with n = 5, p = .5. has a binomial distribution with n = 2, p = .9. *6 A contractor has found through experience that the low bid for a job (excluding his own bid) is a random variable that is Uniformly distributed over the interval (iC, 2C), where C is the contractor's cost estimate (no profit or loss) of the job. If profit is defined as 0 if the contractor does not get the job (his bid is greater than the low bid) and as the difference between his bid and his cost estimate C if he gets the job, what should he bid (in terms of C) in order to maximize his expected profit? 7 A merchant has found that the number of items of brand XYZ that he can sell in a day is a Poisson random variable with mean 4. (a) How many items of brand XYZ should the merchant stock to be 95 percent certain that he wiJ] have enough to last for 25 days? (Give a numerical answer.) (b) What is the expected number of days out of 25 that the merchant will sel1 nO items of brand XYZ? 8 (a) If X is binomia]]y distributed with parameters nand p, what is the distribution of Y=n -- X? (b) Two dice are thrown n times. Let X denote the number of throws in which the number on the first die exceeds the number On the second die. What is the distribution of X? *(c) A drunk performs a" random walk;; over pOsitions 0, 1, 2, ... as fo]]ows: He starts at O. He takes successive one-unit steps, going to the right with probability p and to the left with probabHity 1 - p. His steps are inde(b) (c) (d) (e) X X X X

126

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

Let X denote his position after n steps. Find the distribution of (X + n)/2, and then find 8[X]. *(d) Let Xl (X 2) have a binomial distribution with parameters nand Pl (n and P2)' If Pl <P2, show that P[Xl < k] > P[X2 < k] for k = 0, I, ... , n. (This result says that the smaller the p, the more the binomial distribution is shifted to the left.) 9 In a town with 5000 adu1ts, a sample of 100 is asked their opinion of a proposed municipal project; 60 are found to favor it, and 40 oppose it. If, in fact, the adults of the town were equal1y divided on the proposal, what would be the probability of obtaining a majority of 60 or more favoring it in a sample of 100? 10 A distributor of bean seeds determines from extensive tests that 5 percent of a large batch of seeds will not germinate. He sells the seeds in packages of 200 and gUarantees 90 percent germination. What is the probability that a given package wi1l violate the guar3;ntee? *11 (a) A manufacturing process is intended to produce electrical fuses with no mOre than 1 percent defective. It is checked every hour by trying 10 fuses selected at random from the hour's production. If 1 or more of the 10 fail, the process is halted and carefully examined. If, in fact, its probability of producing a defective fuse is .01, what is the probability that the process will needlessly be examined in a given instance? (b) Referring to part (a), how many fuses (instead of 10) should be tested if the manufacturer desires that the probability be about .95 that the process wi11 be examined when it is producing 10 percent defectives? 12 An insurance company finds that .005 percent of the population die from a certain kind of accident each year. What is the probability that the company must pay off On more than 3 of 10,000 insured risks against such accidents in a given year? 13 (a) If X has a Poisson distribution with P[X = 1] = P[X = 2]. what is P[X = 1 or 2]? (b) If X has a Poisson distribution with mean 1, show that 8[1 X-II] = 2ax/e. *14 Recall Theorems 4 and 8. Formulate, and then prove or disprove a similar theorem for the negative binomial distribution. *15 Let X be normal1y distributed with mean ft and variance a 2 Truncate the density of X on the left at a and On the right at b, and then calculate the mean of the truncated distribution. (Note that the mean of the truncated distribution should fall between a and b. Furthermore, if a = ft - c and b = ft + c, then the mean of the truncated distribution should equal ft.) *16 Show that the hypergeometric distribution can be approximated by the binomial distribution for large M and K; Le., show that

pendent.

PROBLEMS

127

17 Let X be the life in hours of a radio tube. Assume that X is normally distributed with mean 200 and variance a 2 If a purchaser of such radio tubes requires that at least 90 percent of the tubes have lives exceeding 150 hours, what is the largest value a can be and still have the purchaser satisfied? 18 Assume that the number of fatal car accidents in a certain state obeys a Poisson distribution with an average of one per day. (a) What is the probability of more than ten such accidents in a week? (b) What is the probability that more than 2 days will lapse between two such accidents? 19 The distribution given by
I(x; (3) =

1 13

xe-i:(X!P)

2
[(0.

a:(x)

for f3 > 0

20

is cal1ed the Rayleigh distribution. (a) Show that the mean and variance exist, and find them. (b) Does the Rayleigh distribution belong to the Pearsonian system? The distribution given by
I(x; (3)

= fJ3 ..J7T' x 2e- X2 !p 2[(0. OO)(x)

for f3 > 0

21

is ca11ed the Maxwell distribution. (a) Show that the mean and variance exist, and find them. (b) Does this distribution belong to the Pearsonian system? The distribution given by
I(x; n)

= BO,

I
[n _ 2]/2) (1- x

Yn-4)!2[[_1.1l(X)

is called the r distribution. (a) Show that the mean and variance exist, and find them. (b) Does this distribution belong to the Pearsonian system? 22 A die is cast until a 6 appears. What is the probabi1ity that it must be cast more than five times? 23 Red-blood-ce11 deficiency may be determined by examining a specimen of the blood under a microscope. Suppose that a certain small fixed volume contains, on an average, 20 red ceJIs for normal persons. What is the probability that a specimen from a normal person will contain less than 15 red ce]]s? 24 A telephone switchboard handles 600 calls, On an average, during a rush hour. The board can make a maximum of 20 connections per minute. Use the Poisson distribution to evaluate the probability that the board will be overtaxed during any given minute. 25 Suppose that a particle is equa]]y likely to release one, two, or three other particles, and suppose that these secondgeneration particles are in turn each equa]]y likely to release one, two, or three third-generation particles. What is the density of the number of third-generation particles?

128

SPECIAL PARAMETRIC FAMILIES OF UNIVARIATE DISTRIBUTIONS

III

26 Find the mean of the Gumbel distribution. 27 Derive the mean and variance of the Weibul1 distribution. * 28 Show that
P[X > k]

L ~
n (

j=k

pjqn- j =

B(k, n - k

1)

1I"-I( 1 -

1I)"-~ dll

for X a binomial1y distributed random variable. That is, if X is binomial1y distributed with parameters nand p and Y is beta-distributed with parameters k and n - k+ I, then Fy(p) = 1 - Fx(k - I). *29 Suppose that X has a binomial distribution with parameters nand p and Y has a negative binomial distribution with parameters rand p. Show that Fx(r - 1) = 1- Fy(n-r). *30 If U is a random variable that is uniformly distributed over the interval [0, I], then the random variable ZA = [UA - (I - U))']/A is said to have TlIkey's symmetrical lambda distribution. Find the first four moments of Z).. Find two different A's, say .;\1 and ';\2, such that ZAI and Z)'2 have the same first four moments and unit standard deviations.

IV
JOINT AND CONDITIONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE, MORE EXPECTATION

1 INTRODUCTION AND SUMMARY

The purpose of this chapter is to introduce the concepts of k-dimensional distribution functions, conditional distributions, joint and conditional expectation, and independence of random variables. It. like Chap. II, is primarily a " definitions~and~their-understanding" chapter. The chapter is divided into four main sections in addition to the present one. In Sec. 2, joint distributions, both in cumulative and density~function form, are introduced. The important k~dimensional discrete distribution, caned the multinomial, is included as an example. Conditional distributionsand independence of random variables are the subject of Sec. 3. Section 4 deals with expectation with respect to k-variate distributions. Definitions of covariance, the correlation coefficient, and joint moment generating functions, a1l of which are specia.l expectations, are given. The important concept of conditional expectation is discussed in Subsec. 4.3. Results relating independence and expectation are presented in Subsec. 4.5, and the famous Cauchy-Schwarz inequality is proved in Subsec. 4.6. The last main section, Sec. 5, is devoted to the important bivariate normal distribution, which gives one unified example of many of the terms defined in the preceding sections.

130

JOINT AND CONDITIONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

This chapter is the multidimensional analog of Chap. II. It provides definitions needed to understand distributional-theory results of Chap. V.

JOINT DISTRIBUTION FUNCTIONS

In the study of many random experiments, there are, or can be, more than one random variable of interest; hence we are compe11ed to extend our definitions of the distribution and density function of one random variable to those of several random variables. Such definitions are the essence of this section, which is the multivariate counterpart of Secs. 2 and 3 of Chap. II. As in the univariate case we wi]] first define, in Subsec. 2.1, the cumulative distribution function. Although it is not as convenient to work with as density functions, it does exist for any set of k random variables. Density functions for jointly discrete and jointly continuous random variables wi]] be given in Subsecs. 2.2 and 2.3, respectively.

2.1

Cumulative Distribution Function Definition 1 Joint cumulative distribution function Let Xl' X 2 , ,Xk be k random variables a]] defined on the same probability space .(n, .91, P[ D. The joint cumulative distribution function of Xl' "" x k , denoted by Fx" ... ,xJ, ... , .), is defined as P[XI < Xl; ... ; X k '< xd for
a]] (Xl' X2' ... , Xk)'

IIII

Thus a joint cumulative distribution function is a function with domain euclidean k space and counterdomain the interval [0, I]. if k = 2, the joint cumulative distribution function is a function of two variables, and so its domain is just the xy plane.

EXAMPLE 1 Consider the experiment of tossing two tetrahedra (regular four-sided polyhedron) each with sides labeled I to 4. Let X denote the number on the downturned face of the first tetrahedron and Y the larger of the downturned numbers. The goal is to find Fx , y(', .), the joint cumulative distribution function of X and Y. Observe first that the random variables X and Y jointly take on only the values (1, I), (1, 2), (1, 3), (1, 4), (2, 2), (2, 3), (2, 4), (3, 3), (3, 4), (4,4). (The first component is the value of X, and the second the value of Y.)

JOINT DISTRIBUTION FUNGnONS

131

xz

13
~ "0

8 4
3

.. Xl

"2
0 u

=2

~ 1

FIGURE 1 Sample space for experiment of tossing two tetrahedra.

I I I 4 2 3 First tetrahedron

The sample space for this experiment is displayed in Fig.]. The 16 sample points are assumed to be equally likely. Our objective is to find Fx, y(x, y) for each point (x, y). As an example let (x, y) = (2, 3), and find Fx, y(2, 3) = P[X < 2; Y < 3]. Now the event {X < 2 and y.:s;; 3} corresponds to the encircled sample points in Fig. I; hence Fx. y(2, 3) = 6 1 6' Similarly, Fx. y(x, y) can be found for other values of x and y. Fx. y(x, y) is tabled in Fig. 2. IIII

We saw that the cumulative distribution function of a unidimensional random variable had certain properties; the same is true of a joint cumulative. We shaH list these properties for the joint cumulative distribution function of two random variables; the generalization to k dimensions is straightforward.

TABLE OF VALUES OF Fx.r (x,y)

4<y 3<y<4 2<y<3 1<y<2 y<l

0 0 0
0 0

h
1\ -h

.lL

16

-
-h

Ch) 'N
4

-h
N
4

n
-h
0

n
0

-h
0

...l..

16

0
4

....

x<2

2<x<3

3<x<4

FIGURE 2

132

JOINT AND CONDITIONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

Properties of bivariate cumulative distribution function (i)

F( 00,

F( " .)

y) =

lim F(x, y)
X-+ 00

Oforal1 y, F(x, - (0) = lim F(x, y)

Y-+ 00

=0

for all x, and Jim F(x, y)

X-+ 00 Y-+oo

= F( 00,

(0)

= ].

and Yl < Y2, then P[Xl < X < X2; Yl < Y < Y2] = F(X2' Y2) - F(X2' Yl) - F(Xl' Y2) + F(x l , Yl) > O. (iii) F(x, y) is right continuous in each argument; that is, lim F(x + h, y) = Jim F(x, Y + h) = F(x, y). (ii)
X2

If Xl <

o <h-+O

0 <h-+O

We wil1 not prove these properties. Property (ii) is a mono tonicity property of sorts; it is not equivalent to F(Xl' Yl) < F(X2' Y2) for Xl < X2 and Yl < Y2 . Consider, for example, the bivariate function G(x, y) defined as in Fig. 3. Note that G(Xl, Yl) < G(X2' Y2) for Xl < X2 and Yl < Y2' yet G(1 +e,] +e)-G(1 +e,] -e)-G(1-e,1 +e)+G(1-e,] -e)= 1(l - e) - (1 - e) = 2e - I < 0 for e < t; so G(x, y) does not satisfy property (ii) and consequently is not a bivariate cumulative distribution function. Definition 2 Bivariate cumulative distribution function Any function satisfying properties (i) to (iii) is defined to be a bivariate cumulative IIII distribution function without reference to any random variables. Definition 3 Marginal cumulative distribution function IfFx, y( " .) is the joint cumulative distribution function of X and Y, then the cumulative distribution functions Fx (') and F y (') are called marginal cumulative
dis tribution functions.

IIII

TABLE OF G(x, y)

1 <y O<y<l y<O

0 0 0

x
0 0

y
0

x<O

O<x<l

l<x
FIGURE 3

JOINT DISTRIBUTION FUNCTIONS

133

Remark Fx(x) = Fx. y(x, (0), and Fy(y) = Fx. y(oo, y); that is, knowl~ edge of the joint cumulative distribution function of X and Y implies knowledge of the two marginal cumulative distribution functions. fill

The converse of the above remark is not general1y true; in fact, an example (Example 8) will be given in Subsec. 2.3 below that gives an entire family of joint cumulative distribution functions, and each member of the family has the same marginal distributions. We wi11 conclude this section with a remark that gives an inequality inv01ving the joint cumu1ative distribution and marginal distributions. The proof is left as an exercise. Remark
Fx(x)

+ Fy(y) -

I <Fx. y(x, y) <

J Fx(x)Fy(y) for a1l x, y.

fill

2.2

J oint Density Functions for Discrete Random Variables

If Xl' X 2 , , X k are random variables defined on the same probability space, then (Xl' X 2 , , X k ) is cal1ed a k~dimensional random variable.

Definition 4 Joint discrete random variables The k~dimensional ran~ dom variable (XI' X 2' ... , X,J is defined to be a k-dimensional discrete random l'ariable if it can assume values only at a countable number of points (XI' X2, .. , Xk) in k~dimensional real space. We also say that the random variables Xl' X 2 , , X k are joint discrete random variables.

fill
Definition 5 Joint discrete density function If (Xl' X 2 , , X k ) is a k~dimensional discrete random variable, then the jOint discrete density function of (Xl' X 2 , , X k ), denoted byj~l,x2, .... xk(' ., .,., .), is defined to be

for (Xl' X2' otherwise.

... , Xk),

a value of (Xl' X 2 ,

""

X k ) and is defined to be 0

fill
IS

Remark I/x I, xk(x l , " ' , Xk) = 1, where the summation possib1e va1ues of (Xl' ... , X k ).

over all

fill

134

JOINT AND CONDmONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

Ix. y(x, y)

FIGURE 4 x

EXAMPLE 2 Let X denote the number on the downturned face of the first tetrahedron and Ythelargerofthe downturned numbers in the experiment of tossing two tetrahedra. The values that (X, Y) can take on are (I, I), (I , 2), (I, 3), (] , 4), (2, 2), (2, 3), (2, 4), (3, 3), (3, 4), and (4, 4); hence X and Yare jointly discrete. The joint discrete density function of X and Y is given in Fig. 4. In tabular form it is given as
(x, y)
j ~. y(x,y)

(l, ]) (1, 2) (I, 3) (J,4) (2, 2) (2,3) (2,4) (3, 3) (3,4) (4,4)
16
1

16"

16

16

16

16

T6

T6"

T6

16"

or in another tabular form as 4

3
2
]

16 16
T6 T6
1
1
1

16 T6
1

16 16
3

1"6

16

y/x

II/I

Theorem 1 If X and Yare jointly discrete random variables, then knowledge of F x , y(', .) is equivalent to knowledge of Ix. y(', .}. Also, the statement extends to k-dimensional discrete random variables.

JOINT DlsTRIBUOON FUNCTIONS

135'

Let (Xl' Yl)' (X2' Y2)' '" be the possible values of (X, Y). If lx, y(', .) is given, then F x , y(x, y) = I/x, y(Xj, Yi)' where the summa~ tion is over an i for which Xl ::;; X and Yi < y. Conversely, if Fx. y(., .) is given, then for (Xi, Yi), a possible value of (X, Y),
PROOF

Ix,y(X;, Yi) = FX,y(Xb Yi) -

limoFx,y(Xi - h, Y{)
O<h-+

Jim FX.y(Xb Yi - h) o <h-+O Jim Fx,y(xj-h,Yi- h ).

O<h-+O

1111

Definition 6 Marginal discrete density If X and Yare jointly discrete

random variables, then Ix(') and Iy(') are cal1ed marginal discrete density functions. More generally, Jet XiI' ... , Xim be any subset of the jointly discrete random variables Xl' ... , X k ; then IXi I ' Xim(X 11 , , Xim ) is also ca]]ed a marginal density. IIII

Remark If Xl' "', X k are jointly" discrete random variables, then any
marginal discrete density can be found from the joint density, but not conversely. F or example, if X and Yare jointly discrete with values (Xl' YI)' (X2, Y2), ... , then
fX(X k )

U: Xi= Xk}

I Ix. y(Xi' Yi)

and

IIII

Heretofore we have indexed the values of (X, Y) with a single index, namely i. That is, we listed values as (Xl, Yt), (X2 'Y2), ... , (Xi' Yi), ... , The values of (X, Y) could also be indexed by using separate indices for the X and Y values. For instance, we could let i index the possible X values, say X., "" Xi' ... , and j index the possible Y values, say YI' ... , YJ' . . Then the values of (X, Y) would be a subset of the points (Xi' Yj) for i I , 2, ... andj = 1, 2, .... If this latter method of indexing is used, then the marginal density of X is obtained as fo]]ows:
IX(Xk)
=

I/x. y(Xk' YJ)'

j

where the summation is over all Yj for the fixed Xk. The marginal density of Y is analogously obtained. The fo]]owing example may help to c1arify these two different methods of indexing the values of (X, Y).
EXAMPLE 3 Return to the experiment of tossing two tetrahedra, and define X as the number on the downturned face of the first tetrahedron and Yas

the larger of the numbers on the two downturned faces.

The joint

136

JOINT AND CONDITIONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

density of X and Y is given in Fig. 4. The values of (X, Y) can be Hsted as (1, I), (I, 2), (1, 3), (l, 4), (2, 2), (2, 3), (2, 4), (3, 3), (3, 4), and (4, 4), 10 points in an. Or, if we note that X has values I, 2, 3, and 4; Y has values I, 2, 3, and 4; and Y is greater than or equal X, the values of (X, Y) are {(i,j): i = 1, ... ,4; j = 1, ... ,4; and i <j}. Let us use each of these methods of indexing to evaluate Fx. y(2, 3) from the joint density. Under the first method of indexing,

Fx, y(2, 3)

{i:Xi::;; 2. )Ii::;; 3}

fx. y(X;, Yi)

=fx,y(l, 1)+ lx, y(l, 2)

+ lx, y(1, 3) + Ix.

y(2, 2) + Ix. y(2, 3)

10

Under the second method of indexing,

F x ,y(2, 3)

=I

Ifx,y(i,j) =

6 1 6'

i=lj=i

Similarly, an other values of Fx. y(., .) could be obtained.

ly(3)

Also
3)

= I
{i:y=3}

Ix, y(Xj, Yi) = lx, y(1, 3)

3 16

+ Ix,

y(2, 3)

+ lx, y(3,

= l6 + rt +

5 1 6'

Simi1arly Iy(l) = -n" fy(2) = -hi, and ly(4) = 176' which together with ly(3) = 156 give the marginal discrete density function of Y. 1111

EXAMPLE 4 We mentioned that marginal densities can be obtained from the joint density, but not conversely. The following is an example of a family of joi~t densities that a)) have the same marginals, and hence we see that in genera] the joint density is not uniquely determined from knowledge of the marginals. Consider altering the joint density given in the previous examples as follows: 4 3
16
1

+e
e

/6 -

TI

TI

/6 TI
1

rt+e
16
2

1"6

2
1

16

Y/x

JOINT DISTRIBUTION FUNCTIONS

137

For each 0 < e < /6' the above table defines a joint density. Note that the marginal densities are independent of e, and hence each of the joint densities (there is a different joint density for each 0 < e < -16) has the same marginals. IIII We saw that the binomial distribution was associated with independent, repeated Bernoul1i trials; we shaH see in the example below that the multinomial distribution is associated with independent, repeated trials that generalize from Bernoul1i trials with two outcomes to more than two outcomes.

EXAMPLE 5 Suppose that there are k + ] (distinct) possible outcomes of a trial. Denote these 0 utcomes by J I' J2, ... , jk + I' and let Pi = P[Jd,

i = ], ... , k + I.

k+l

Obviously we must have

Pi

= I, just as P + q = I in

i= 1

the binomial case. Suppose that we repeat the trial n times. Let Xi denote the number of times outcome .J i occurs in the n trials, i = I, "', k + I. Jf the trials are repeated and independent, then the discrete density function of the random variables Xl' ... , X k is

k+ I

wherexj=O, ... ,nand

Xj=n.

NotethatXk+1=n-

I
i

Xj'
I

i= I

To justify Eq. (I), note that the left-hand side is P[X1 = XI; X 2 = X2; ... ; X k + I Xk+ d; so, we want the probability that the n trials result in exactly Xl outcomes JI' exactly X2 outcomes "2, .. exactly Xk+ I outcomes
k+1

,Jk+l'

where

II

Xi

n.

Any specific ordering of these

outcomes has

probability p~l . p~2 ... Pk~\l by the assumption of independent trials. and there are n!/xt! x 2 ! ... Xk+l! such orderings. IIII
Definition 7 Multinomial distribution The joint discrete density function given in Eq. (1) is called the multinomial distribution. IIII

The multinomial distribution is a (k + I) parameter family of distributions, the parameters being n and PI' P2, "" Pk' Pk + 1 is, like q in the binomial distribution, exactly determined by Pk+ t = I - PI - P2 - ... - Pk' A

138

JOINT AND CONDITIONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

.20

FIGURE 5

Xl

particular case of a multinomial distribution is 0 btained by putting, for example, n = 3, k = 2, Pl = .2, and P2 = .3, to get
!Xl,X2(Xl' X2)

=f(x l ,

X2) =

Xl ,X2

'(3 ~

3'
Xl

X2'

)' (.2Yl(.3YZ(.5)3

This density is plotted in Fig. 5. We might observe that if Xl' X 2 , , X k have the multinomial distribution given in Eq. (1), then the marginal distribution of Xi is a binomiaJ distribution with parameters n and Pi' This observation can be verified by recalJing the experiment of repeated, independent trials. Each trial can be thought of as resuJting either in outcome 6i or not in outcome 6j, in which case the trial is Bernoul1i, implying that Xi has a binomial distribution with parameters n and Pi'

2.3

Joint Density Functions for Continuous Random Variables Definition 8 Joint continuous random variables and density function The
k-dimensional random variable (Xl' X 2, ... , X k ) is defined to be a k-dimensional continuous random variable if and only if there exists a function fXI .... Xk(', , .) ~ 0 such that
FXt. .... Xk(X l ' ""Xk)=

J~koo '"

J:loofxh ... ,Xk(Ut"",Uk)dul

... , .)

. duk

(2)

for all (Xl' ... , Xk)' density function.

fXI ..... Xk('

is defined to be a joint probability

IIII

JOINT DISTRIBUTION FUNCTIONS

139

As in the unidimensional case, a joint probability density function has two properties: (i)
(ii)

fx

1 ,

roo'" r

,Xk (X 1 , " ' , Xk)

> O.
... ,

/x ...... x,(x 1 ,

x.) dx, ... dx,

= 1.

A unidimensional probability density function was used to find probabilities. For example, for X a continuous random variable with probability density Ix('), P[a < X < b] = J!/x(x) dx; that is, the area under Ix(') over the interval (a, b) gave P[a < X < b]; and, more genera]]y, P[X E B] = JBlx(x) dx; that is, the area under fx(') over the set B gave P[X E B]. In the two-dimensional case, volume gives probabilities. For instance, let (Xl' X 2) be jointly continuous random variables with joint probability density function lXI, x2(x 1, x 2), and let R be some region in the X1X2 plane; then P[(X1' X 2) E R] = JJfXI,X2(X1' x 2) dx 1 dX2; that is, the probability that (Xl' X 2) fa]]s in the
R

region R is given by the volume under IXI. X2(', .) over the region R. lar if R = {(Xl' X2): a 1 < Xl < h1; a2 < X2 < b2}, then

In particu-

Ajoint probabllitydensityfunction is defined as any nonnegative integrand satisfying Eq. (2) and hence is not uniquely defined.

EXAMPLE 6

Consider the bivariate function

f(x, y) = K(x

+ y)I(o, l)(x)/(O, 1)(Y) = K(x + y)/u(x, y),

where U = {(x, y): 0 < x < ] and 0 < y < I}, a unit square. Can the constant K be selected so that f(x, y) wi]] be a joint probability density function? If K is positive, I(x, y) > O.

5_

00

00

f-

00

oo

Kf(x, y) dx dy

fo

t
1

K(x
1

+ y) dx dy

=K
= K

I I (x + y) dx dy

I (t + y) dy
1

=K(t+-D
=1

140

JOINT AND CONDITIONAL DlSTRmUTlONS, STOCHASTIC INDEPENDENCE

IV

f(x, y)

(1, 1, 2)

(1,0, 1)

(0, 1, 1)
--~--~~------~--x

/ _____ ....Y____ _

I // I /

FIGURE 6

for K = 1. So lex, y) = (x + y)I(o, l)(x)I(o, 1)(Y) is a joint probability density function. It is sketched in Fig. 6. Probabilities of events defined in terms of the random variables can be obtained by integrating the joint probability density function over the indicated region; for example
t t

P[O<X<t;O<Y<:!-]= f
=
_ -

f(x+y)dxdy

: (1 +~)
1 32

dy

+ 64 1 + y over the region

{(x, y):

-2-

64'

which is the volume under the surface z = x 0< x < t; 0 < y<!} in the xy plane.

///1

Theorem 2 If X and Yare jointly continuous random variables, then

knowledge of Fx. y(., .) is equivalent to knowledge of an/x, y(., .). remark extends to k~dimensional continuous random variables.
PROOF

The

For a given /x y(., .), Fx y(x, y) is obtained for any

(x, y) by

JOINT DlsTRmUTION FUNCTIONS

141

For given Fx,Y(', .), anfx,Y(x, y) can be obtained by

fx, y(x, y)

iJ2 Fx y(x, y) iJ~ iJy

for x, y points, where Fx, y(x, y) is differentiable.

IIII

Definition 9 Marginal probability density functions If X and Yare jointly continuous random variables, then fx(') and fy(') are ca]]ed marginal probability density functions. More general1y, let XiI' ... , Xim be any subset of the jointly continuous random variables Xl' "', X k IXi!' "', Xim (xiI' ... , x im ) is ca]]ed a marginal density of the m-dimensional

random variable (XiI' ... , Xi m)'

IIII

Remark If Xl' .. " X k are jointly continuous random variables, then any marginal probability density function can be found. (However, knowledge of a11 marginal densities does not, in general, imply knowledge of the joint density, as Example 8 below shows.) If X and Yare jointly continuous, then
fx(x) =

f~ oofx, y(x, y) dy

and

(3)

smce
Ix(x)

dF x(x) dx

= dx f_

d [X
00

(fOO
_

) ] oofx, y(u, y) dy du

= f _ oofx, y(x, y) dy.

00

I1I1
EXAMPLE 7 Consider the joint probability density
fx, y(x, y) Fx,Y(x, y)

= (x + y)l(o, l)(x)l(o, 1)(Y)' = l(o,l)(x)l(o,l)(Y)

f f (u + v) du dv
y

+ 1(0, l)(x)l[l, OO)(y) fo + 1[1, oo)(x)I(o, l)(Y) fo

=
y

fo (u
1

+ v) du + v) du

dv dv

fo (u

+ l[l, oo)(x)I[1, ooly) 2 t{(x y + xy2)1(0.1)(x)I(0, l)(Y) + (x 2 + x)l(o, l)(x)I[1, oo)(y) + (y + y2)1[1, oo)(x)l(o, l)(Y)} + 1[1, oo)(x)I[1, OO)(y).

142

JOINT AND CONDITIONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

fx(x) =

00

fx, y(X, y) dy

-00

= 1(0, l)(X)

f (X + y) dy
1

= (X + !)/(o, l)(X);
or,
fx(x) = aFx,y(x, oo) ax aF x(X) ax

a = l(o,ll x ) ax

(+x) 2
IIII

= (x + !-)/(o, l)(X).

EXAMPLE 8 Let /x(x) and /y(y) be two probability density functions with corresponding cumulative distribution functions Fx(x) and Fy(y), respectively. For - I < ex < I, define
Ix, y(x, y; ex) = /x(x)/y(y}{1

+ ex[ 2Fx(x) - 1][2Fy(Y)

- In.

(4)

We will show (i) that for each ex satisfying -I < ex < I, fx, y(x, y; ex) is a joint probability density function and (ii) that the marginals of/x, y(x, y; ex) are/x(x) and/y(y), respectively. Thus, {Ix, y(x, y; ex): -I < ex < I} will be an infinite family of joint probability density functions, each having the same two given marginals. To verify (i) we must show that/x , y(x, y; ex) is nonnegative and, if integrated over the xy plane, integrates to I.
/x(x)fy(y}{l

+ ex[2F x(x) - 1][2Fy(y)

- I]}

>0

if I >-ex[2Fx(x) - 1][2Fy(y) - I]; but ex, 2Fx(x) - I, and 2Fy(y) - I are all between -I and I, and hence also their product, which implies/x. y(x, y; ex) is nonnegative. Since

r/x(X) dx

f'oo (roo fx.

y(x, y; rx) dy ) dx,

CONDITIONAL DISTRIBUTIONS AND STOCHASTIC INDEPENDENCE

143

it suffices to show that/x(x) and/y(y) are the marginals of/x, y(x, y; ex).

00

fx, y(x, y; ex) dy fx<x)fY(Y}{1

00

-00

= foo

+ ex[2F x(x) -

1][2F y(y) - I]} dy -

-00

=/x(x) I-fY(Y) dy = fx(x),

+ ex/x(x)[2Fx(x)

l]f_

00

[2F y(y) - l]fy(y) dy

00

noting that
1) du = 0

f_

00
00

[2F y (y) - l]fy(y) dy

fo (2u 1

by making the transformation u = Fy(y).

IIII

CONDITIONAL DISTRIBUTIONS AND STOCHASTIC INDEPENDENCE

] n the preceding section we defined the joint distribution and joint density functions of several random variables; in this section we define conditional distributions and the related concept of stochastic independence. Most definitions will be given first for only two random variables and later extended to k random variables.

3.1

Conditional Distribution Functions for Discrete Random Variables Definition 10 Conditional discrete density function Let X and Y be jointly discrete random variables with joint discrete density function Ix, y(., .). The conditional discrete density function of Y given X = x, denoted by /Ylx(1 x), is defined to be

f
YIX

(Ix)_/x,y(x,y) y /x(x) ,

(5)

if fx(x) > 0, where fx(x) is the marginal density of X evaluated at x. /Ylx( Ix) is undefined for fx(x) = O. Similarly,

f
if fy(y) >

xlr

(xl ) _ Ix, y(x, y) Y fy(y) ,

(6)

o.

IIII

144

JOINT AND CONDmONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

Since X and Yare discrete, they have mass points, say Xl' X2, ... for X and YI, Y2' for Y. If Ix(x) > 0, then X = Xi for some i, and IX(Xi) = P[X = xJ The numerator of the right-hand side of Eq. (5) is lx, y(Xi' J'j) = P[X = Xi; Y = Yj]; so

for Yl a mass point of Yand Xi a mass point of X; hence !Ylx(" Ix) is a conditional probability as defined in Subsec. 3.6 of Chap. l. !Ylx( Ix) is called a conditional discrete density function and hence should possess the properties of a discrete density function. To see that it does, consider X as some fixed mass point of X. Then IYlx(yl x) is a function with argument Y; and to be a discrete density function must be nonnegative and, if summed over the possible values (mass points) of Y, must sum to 1. IYlx(yl x) is nonnegative since !x, y(x, y) is nonnegative and Ix(x) is positive.

" L-fYlx(y1Ix) 1

= Lj

"Ix. y(x, Yj) 1" fx(x) j () =j-() 4- fx,y(x, Yj) =j-( ) x x xX J xX

= 1,

where the summation is over all the mass points of Y. (We used the fact that the marginal discrete density of X is obtained by summing the joint density of X and Y over the possible values of Y.) So !Ylx(" Ix) is indeed a density; it tells us how the values of Yare distributed for a given value x of X. The conditional cumulative distribution of Y given X = x can be defined for two jointly discrete random variables by recalling the close relationship between discrete density functions and cumulative distribution functions. Definition 11 Conditional discrete cumulative distribution If X and Y are jointly discrete random variables, the conditional cumulative distribution of Y given X = x, denoted by F y1x( Ix), is defined to be Fy1x(Y1 x) = P[ Y < YI X = x] for Ix(x) > o. IIII

Remark Fy1x(Y1 x) =

L
{j:Yj:;;;Y}

IYlx(Yjl x).

IIII

EXAMPLE 9 Return to the experiment of tossing two tetrahedra. Let X denote the number on the downturned face of the first and Y the larger of the downturned numbers. What is the density of Y given that X = 2?

CONDITIONAL DISTRIBUTIONS AND STOCHASTIC INDEPENDENCE

145

ly/x(21 2) =

Ix Ix

y(2, 2) ix(2)

-(6
=

T\ = 2
=4
1
1

IYlx(312) =
ly/x(41 2) =

y(2, 3) rt ix(2) = T~ y(2,4) ix(2)

Ix

n = 4'
IIII

l6

Also, for y = 3 for y = 4.

Definition 12 Conditional discrete density function Let (Xl' ... , Xk ) be a k-dimensional discrete random variable, and let XII"'" Xi .. and Xii' ... , Xjs be two disjoint subsets of the random variables Xb ... , X k The conditional density of the r-dimensional random variable (Xi I ' , XiJ given the value (x j l ' . . . , x)J of (Xj l ' " X j ) is defined to be

_lxil' .... xi,.,Xh .. ,XJ"'(Xil' , Xi,., Xii' ""Xis)

IXh .... ,

xJ",(xit'

.. " X j,.}

IIII

EXAMPLE 10 Let Xl' ... , Xs be jointly discrete random variables, Take r = s = 2, (Xii' X i2 ) = (Xt, X 2 ), and (XiI' X h ) = (X3' Xs); then
JXl,X2IXJ,XsXbX2X3,XS-

(I

)-

lXI, X2, XJ. xs(XI, X2, Xl' Xs)

I" J XJ.

Xs X 3 , Xs

IIII

EXAMPLE II Suppose 12 cards are drawn without replacement from an ordinary deck of playing cards. Let Xl be the number of aces drawn, X 2 be the number of 2s, X3 be the number of 3s, and X 4 be the number of 4s. The joint density of these four random variables is given by

146

JOINT AND CONDITIONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

where Xi = 0, I, 2, 3, or 4 and i = I, ... ,4, subject to the restriction that LXi < 12. There-are a large number of conditional densities associated with this density; an example is
!X2,X4Ix 1 ,xJ (X2, x 4 1x h
X3)

(12 - x~~x,)
where 3.2
Xi

= 0,

1, .. " 4 and

X2

+ X 4 < 12 -

Xl -

X3'

IIII

Conditional Distribution Functions for Continuous Random Variables Definition 13 Conditional probability- density function Let X and y be jointly continuous random variables with joint probability density function Ix. y(x, y). The conditional probability density junction of Y given X = X, denoted by IYlx('1 x), is defined to be
I' ( 1 ) _ JYIX Y X -

Ix. y(x, y) /x(x)

(7)

if Ix(x) > 0, where Ix(x) is the marginal probability density of X, and is undefined at points when Ix(x) = O. Similarly,
IXly(xly) = y(x, y) Iy(y)
if/y(Y) > 0,

(8)

and is undefined if /y(y) = 0

IIII

IYlx( -I x) is called a (conditional) probability density function and hence

should possess the properties of a probability density function. IYlx( -I x) is clearly nonnegative, and

CONDmONAL DISTRIBUTIONS AND STOCHASTIC INDEPENDENCE

147

The density IYlx(1 x) is a density of the random variable Y given that x is the value of the random variable X. In the conditional density IYlx( J x), x is fixed and could be thought of as a parameter. Consider IYlx('1 xo), that is, the density of Y given that X was observed to be Xo Now Ix, y(x, y) plots as a surface over the xy plane. A plane perpendicular to the xy plane which intersects the xy plane on the line x = Xo will intersect the surface in the curve Ix. y(xo, y). The area under this curve is
00

f i x . y(Xo, y) dy = Ix(xo)
-00

Hence, if we divide Ix. y(Xo , y) by Ix(xo), we obtain a density which is precisely /Ylx(yl xo) Again, the conditional cumulative distribution can be defined in the natural way.

Definition 14 Conditional continuous cumulative distribution If X and Yare jointly continuous, then the conditional cumulative distribution of Y gi ven X = x is defined as
Fy1x(yl x) = fY IYlx(zl x) dz
-00

for all x such that/x(x) > O.

IIII

EXAMPLE 12 r

Suppose/x. y(x, y) = (x

+ y)I(o, l)(x)I(o, 1)(Y)

x
X

(I ) Y x -

(x

+ y)l(o, Il x )I(o, 1)(Y) _

(+ X
1)1 () 2: (0,1) X -

+Y 1
+ 1: 1
(0.1)

JYIX

() Y

for 0 < x < I.

Note that Fy1x(ylx)

=

f~oo/Ylx(zlx) dz

fox+t
I
--1

yx

+z

dz

fY
0

x+t

(x

+ z) dz
IIII

x+'!

(xy

+ y2/2)

for 0 < y < I.

Conditional probability density functions can be analogously defined for k-dimensional continuous random variables. F or instance,

f Xl,X2.X

Ixl,XS

(X l' x 2, x 4 I X3,

Xs

)_fXl,X2,Xl,X4,Xs(Xl,X2,X3,X4'XS)
-

for /Xl' xs( X3 , x 5) >

o.

Xl,X s

x3

Xs

148

JOINT AND CONDITIONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

3.3

More on Conditional Distribution Functions

We have defined the conditional cumulative distribution Fy1x(yl x) for either jointly continuous or jointly discrete random variables. If X is discrete and Y is any random variable, then Fy1x(Ylx) can be defined as PlY yl X = xl if x is a mass point of X. We would like to define PlY ~ yl X = x] and more generally P[A I X = x], where A is any event, for X either a discrete or continuous random variable. Thus we seek to define the conditional probability of an
event A given a random variable X = x.

We start by assuming that the event A and the random variable X are both defined on the same probability space. We want to define P[A I X = x]. If X is discrete, either x is a mass point of X, or it is not; and if x is a mass point of X,
P[A I X

= 1=
x

P[A; X = xl P[X = xl '

which is well defined; on the other hand, if x is not a mass point of X, we are not interested in P[A I X = xl. Now if X is continuous, P[A I X = xl cannot be analogously defined since P[X = xl = 0; however, if x is such that the events {x - h < X < x + h} have positive probabllity for every h > 0, then P[A I X = x] could be defined as
P[A I X =

xl =

o <h-+O

lim P[A Ix - h < X <

x + hl

(9)

provided that the limit exists. We will take Eq. (9) as our definition of P[A I X = xl if the indicated limit exists, and leave P[A I X = xl undefined otherwise. (It is, in fact, possible to give P[A I X = xl meaning even'if P[X = x] = 0, and such is done in advanced probability theory.) We will seldom be interested in P[A I X = x] per se, but will be interested in using it to calculate certain probabilities. We note the following formulas:
I

(i)

P[A]

=I

00

i= 1

P[A I X

= xdfx(Xi)

(10)

if X is discrete with mass points (ii) if X is continuous. (iii)

P[A; X
E

Xl' X2,

P[A] =

-00

00

P[A I X = x]fx(x) dx

(11)

B] =

P[A I X = xdfx(Xi)

(12)

{i:Xf eB}

CONDITIONAL DISTRIBUTIONS AND STOCHASTIC INDEPENDENCE

149

if X is discrete with mass points (iv) if X is continuous.

P[A; X
E

Xl' X2, ..

B] =

f P[A I X
B

x]fx(x) dx

(13)

Ai' .;gh we will nQt prove the above formulas, we note that Eq. (10) is just tl:-:' iheorem of total probabilities given in Subsec. 3.6 of Chap. I and the other& are generalizations of the same. Some problems are of such a nature that it is easy to find P[A I X = x] and difficult to find P[A]. If, however, /x( .) is known, then PtA] can be easily obtained using the appropriate one of the above formulas. Remark Fx. y(x, y) = S: ooFy1x(yl x')fx(x') dx' results from Eq. (13) by taking A = {Y < y} and B = (- 00, x]; and Fy(y) = J~ooFYlx(yl x)/x(x) dx is obtained from Eq. (II) by taking A = {Y < y}. IIII We add one other formula, whose proof is also omitted. Suppose A = {h( X, Y) < z}, where h( " .) is some function of two variables; then
(v) P[A I X

= x] = P[h(X,

Y)

zl X

= x] = P[h(x,

Y} < zl X

= x].
(14)

The following is a classical example that uses Eq. (II); another example utilizing Eqs. (14) and (II) appears at the end of the next subsection.

EXAMPLE 13 Three points are selected randomly on the circumference of a circle. What is the probability that there will be a semicircle on which all three points will lie? By selecting a point" randomly," we mean that the point is equally likely to he any point on the circumference of the circle; that is, the point is uniformly distributed over the circumference of the circle. Let us use the first point to orient the circle; for example, orient the circle (assumed centered at the origin) sO that the first point falls on the positive x axis. Let X denote the position of the second point, and let A denote the event that all three points lie on the same half circle. X is uniformly distributed over the interva1 (0, 2n). According to Eq. (II), P[A] = SP[A I X = x]/x(x) dx. Note that for 0 < x < n, P[A I X = xl = (n - x + n)/2n since, given X = x, event A occurs if and only if the third point falls between x - nand n. Similarly, P[A I X = x] = (x + n - n)/2n for n x < 2n. Hence P[A] = J~1t P[A I X = x](l/2n) dx =
(l/2n){jo[(2n - x)/2n] dx

+ J;1t(x/2n) dx}=,f.

II1I

150

JOINT AND CONDITIONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

3.4 .Independence
When we defined the conditional probability of two events in Chap. I, we also defined independence of events. We have now defined the conditional distribution of random variables; so we should define independence of random variables as well.

Definition 15 Stochastic independence Let (Xl' X 2, ... , X k ) be a k-dimensional random variable. X., X 2 , , X k are defined to be stochastically independent if and only if
F Xl ..... X,,(XI' .. , Xk)

1:= I

n F XlXI)

(15)

Definition 16 Stochastic independence Let (Xl, X 2, .. , Xk ) be a k-dimensional discrete random variable with joint discrete density function fx 1. x,,( " ... , '). Xl" .. , X k are stochastically independent if and only if
k

fXl ..... X"(X., .. " Xk) = nfxlxt)

j=

(16)

Definition 17 Stochastic independence Let (Xl' "', X k ) be a k-dimensional continuous random variable with joint probability density function fx 1.... , x,,( " ... , '). Xl" .. , X k are stochastically independent if and only if
k

fXI ..... X"(X., ... , Xk)

nfxlxi)
i= 1

(17)

forallxt"",xk'

IIII IIII

Remark Often the word "stochastically" will be omitted.

We saw that independence of events was closely related to conditional probability; likewise independence of random variables is closely related to conditional distributions of random variables. For example, suppose X and Y are two independent random variables; thenfx, y(x, y) =/x{x)fy(y) by definition of independence; however, fx, y(x, y) = fy[x(yl x)fX<x) by definition of conditional density, which implies thatfYlx(Y1 x) = fy(y); that is, the conditional

CONDmONAL DISTRIBUTIONS AND STOCHASTIC INDEPENDENCE

151

density of Y given x is the unconditional density of Y. So to show that two random variables are not independent, it suffices to show thatfYjx(yl x) depends on x.

EXAMPLE 14 Let X be the number on the downturned face of the first tetrahedron and Y the larger of the two downturned numbers in the experiment of tossing two tetrahedra. Are X and Y independent? Ob3 viouslynot, since/Y lx(21 3) = P[ Y = 21 X = 3] = 0 =I: fy(2) = P[ Y = 2] = 1 6' IIII

EXAMPLE J 5 Let Ix. y(x, y) = (x + y)I(O.l)(x)I(o.l)(Y)' Are therefore X and Y independent? No, since fYjx(Y 1 x) = [(x + y)/(x + !)]I(o.l)(Y) for 0< x < l,fYlx(ylx) depends on x and hence cannot equal/y(y). IIII

EXAMPLE 16 Let Ix. y(x, y) = e-(x+Y)I(o. oo)(x)I(o. oo)(Y). independent since

fx. y( x, y)
=

and

Yare

[e-xI(o. oo)(x)][e- YI(o. OO)(Y)]

fx(x)fy(y) IIII

for all (x, y).

It can be proved that if XI. ... , X k are jointly continuous random variables,

then Definitions IS and 17 are equivalent. Similarly, for jointly discrete random variables, Definitions 15 and 16 are equivalent. It can also be proved. that Eq. (15) is equivalent to P[XI
Bl ,
,

B l ; ... ; X k E

Bd =

i= 1

n P[X,
k

B i ] for sets

The following important result is easily derived using the above equivalent notions of independence.
Bk

Theorem 3 If Xl' ... , X k are independent random variables and gl('), "', gk(') are k functions such that Y j = giXj)' .i = I, ... , k are random variables, then Yl , ... , Yk are independent.
Note that if gjl(Bj ) = {z: gj(z) E B), then the events { Y j E B j } and {Xj E gjl(Bj)} are equivalent; consequently, P[ Yl E B l ; ... ; k Y E Bd = P[X E gl l (B ); . . . X E g,;-l(Bk)] = P[Xj E gjl(B)]
PROOF
k I 1 k

j=l

" P[Yj n

j= 1

Bj ].

IIII

152

JelNT AND CONDITIONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

For k = 2, the above theorem states that if two random variables, say X and Y, are independent, then a function of X is independent of a function of Y. Such a result is certainly intuitively plausible. We will return to independence of random variables in S ubsec. 4.5. Equation (14) of the previous subsection states that P[h(X, Y) < zl X = x] = P[h(x, y) < z I X = x]. Now if X and Y are assumed to be independent, then P[h(x, y) < zl X = x] = P[h(x, y) < z], which is a probability that may be easy to calculate for certain problems.

EXAMPLE 17 Let a random variable Y represent the diameter of a shaft and a random variable X represent the inside diameter of the housing that is intended to support the shaft. By design the shaft is to have diameter 99.5 units and the housing inside diameter 100 units. If the manufacturing process of each of the items is imperfect, so that in fact Y is uniformly distributed over the interval (98.5, 100.5) and X is uniformly distributed over (99, 101), what is the probability that a particular shaft can be successfully paired with a particular housing, when" successfully paired" is taken to mean that X - h < Y < X for some small positive quantity h? Assume that X and Yare independent; then
P[X - h < Y < X] =
=

foo

P[X - h < Y < Xl X = x]/x(x) dx P[x - h < Y < x]tdx.

-00

101

99

Suppose now that h = 1; then

x - 98.5

2
P[x - 1 < Y < x]

for 99 < x < 99.5 for 99.5 < x < 100.5 for 100.5 < x < 101.

= -

I 2

100.5 - (x - 1)

2
Hence,
P[X - 1 < Y <

Xl =

f =f

101

P[x - 1 < Y < x]t dx

99 99.5

1(X - 98.5)! dx

99

100.5

t(t) dx +

99.5

101

(t)(100.5 - x + l)t dx =

7 1 6.

100.5

IIII

EXPECTATION

153

4 EXPECT A TION
When we introduced the concept of expectation for univariate random variables in Sec. 4 of Chap. II, we first defined the mean and variance as particular expectations and then defined the expectation of a general function of a random variable. Here, we will commence, in Subsec. 4.1, with the definition of the expectation of a general function of a k-dimensional random variable. The definition will be given for only those k-dimensional random variables which ha ve densities.

4.1

Definition Definition 18 Expectation Let (Xl, ... , X k ) be a k-dimensional random variable with density IXI .... Xk(', ... , '). The expected value of a function g(', ... , .) of the k-dimensional random variable, denoted by 8[g(X1> "', X,,)], is defined to be
8[g(X1> ... , Xk)]

= L g(Xb ... , xk)/xl . .... X (Xl' .. "

k

x,,)

(18)

if the random variable (Xl' ... , Xk ) is discrete where the summation is over all possible values of (Xh ... , X k ), and

(Xl (Xl (Xl = f_(Xl f_(Xl ... f_(Xl g(Xl'

... , x l )!XI .... Xk (X 1 '

.. ,

xk ) dX 1

dXk

(19)

if the random variable (Xl' ... , X,,) is continuous.

///1

In order for the above to be defined, it is understood that the sum and mUltiple integral, respectively, exist.

Theorem 4 In particular, if g(Xl' ... , Xk) = Xi' then

8[g(Xl' "" X,,)] = 8[Xd = /lx,.
(20)

Assume that (Xl, ... , X k ) is continuous. (Xl' ... , Xk) discrete is similar.]
PROOF

[The proof for

8[g(Xl' ... , X k )] =
=

(Xl (Xl J-(Xl f_(Xl '" f_(Xl xi!xt, ... ,Xk(Xh ... , Xk) dXl
(Xl J(Xl -(Xl xdx.(xt) dXr = 8[X,]

... dx"

154

JOINT AND CONDmONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

using the fact that the marginal density /XiXi) is obtained from the joint density by

f f
00 -00

00

fXl, ... ,Xk(X1,,Xk)dx1dxi-1dxi+1.dxk

IIII

-00

Similarly, the following theorem can be proved.

Theorem 5 If g(X1, ... , Xk) = (Xi - 8[Xd)2, then

8[g(Xb ... , X k)]

= 8[(Xi

8[Xi])2]

= var [Xi].

IIII

We might note that the" expectation" in the notation 8[Xi ] of Eq. (20) has two different interpretations; one is that the expectation is taken over the joint distribution of Xl' ... , X k , and the other is that the expectation is taken over the marginal distribution of Xl. What Theorem 4 really says is that these two expectations are equivalent, and hence we are justified in using the same notation for both.

EXA MPLE 18 Consider the experiment of tossing two tetrahedra. Let X be the number on the first and Y the larger of the two numbers. We gave the joint discrete density function of X and Y in Example 2.
8[XY] =

xy/x, y(X, y)

= 1 . 1(/6) + 1 2(/6) + 1 . 3(ft) + 1 4(/6)

+ 22(/6) + 2 . 3(/6) + 2 . 4(/6) + 3 . 3(/6) + 3 . 4(-Ar) + 4 . 4(146) = \365. 8[X + y] = (1 + 1)/6 + (1 + 2)/6 + (1 + 3)/6 + (l + 4)-Ar + (2 + 2)l6 + (2 + 3)-Ar + (2 + 4)-Ar + (3 + 3)-h 4 + (3 + 4) /6 + (4 + 4) 1 6 = i ~. 8[X] = i, and 8[y] = ~~; hence 8[X + y] = 8[X] + 8[Y].
EXAMPLE 19 Suppose lx, y(x, y) = (x
1

IIII

+ y)I(o,1)(x)I(o, l)(Y)
1

8[XY] 8[X

f f
1

xy(x
1

+ y) dx dy =-1-.

+ Y] =

tt

(x

+ y)( x + y) dx d y = ~.
7 1 2.

8[X] = 8[Y] =

IIII

EXPECTATION

155

EXAMPLE 20 Let the three-dimensional random variable (Xl' X 2 , X 3) have the density
fXl.X2,xixh X2, X3)
= 8XIX2

X3 I(o.l)(xd I (o, 1)(x2)I(o,l)(X3)'

Suppose we want to find (i) 8[3XI + 2X2 + 6X3], (ii) 8[XI X 2X 3], and (iii) 8[XI X 2]. For (i) we have g(Xh X2, X3) = 3XI + 2X2 + 6X3 and obtain
8[g(XI , X 2 , X 3)]
=

=
For (ii), we get

nnn

8[3XI

+ 2X2 + 6X3] (3XI + 2x2 + 6X3)8xIX2 X3 dXI dX2 dX3

2
8 2 "

= \2.

8[XI X 2 X 3 ]

nJ~ J5 8X/X2 x/ dXI dX2 dX3 =

=l

and for (iii) we get 8[XI X 2 ]

IIII

The following remark, the proof of which is left to the reader, displays a property of joint expectation. It is a generalization of (ii) in Theorem 3 of Chap. II.

Remark

8[~ CiUi(Xl . . . . XJ] = ~ ci 8[u,(X" ... , X.)l

for constants

CI 'C 2 ',C m

1/11

4.2

Covariance and Correlation Coefficient Definition 19 . Covariance Let X and Y be any two random variables defined on the same probability space. The covariance of X and Y, denoted by cov [X, Y] or (Ix, y, is defined as
COv

[X, Y]

= 8[(X - ttx)(Y-tty)]

(21)

provided that the indicated expectation exists.

III/

Definition 20 Correlation coefficient The correlation coefficient, denoted by p[X, Y] or Px, y, of random variables X and Y is defined to be
px , y =
COv

[X, Y] (Ix (Iy

(22)

provided that cov [X~ y], (Ix, and (Iy exist, and (Ix > 0 and (Iy > O.

III I

156

JOINT AND CQNDmONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

Both the covariance and the correlation coefficient of random variables X and Yare measures of a linear relationship of X and Y in the following sense: cov [X, Y] will be positive when X - J1x and Y - J1y tend to have the same sign with high probability, and cov [X, Y] will be negative when X - Jlx and Y - Jly tend to have opposite signs with high probability. cov [X, Y] tends to measure the linear relationship of X and Y; however, its actual magnitude does not have much meaning since it depends on the variability of X and Y. The correlation coefficient removes, in a sense, the individual variability of each X and Y by dividing the covariance by the product of the standard deviations, and thus the correlation coefficient is a better measure of the linear relationship of X and Y than is the covariance. Also, the correlation coefficient is unitless and, as we shall see in Subsec. 4.6 below, satisfies - I Px , y :s: I. Remark cov lX, Y]
PROOF

= 4[(X - J1x)(Y - J1y)] = 4[XY] - J1xJ1Y

4[(X - J1x)( Y - J1y)]

= 4[XY - J1x Y - J1y X + J1x J1y] = 4[XY] - J1x4[ Y] - J1y 4[X] + J1xJly
= 4[XY] - J1x J1y.

1111

EXAMPLE 21 Find Px , y for X, the number on the first, and Y, the larger of the two numbers, in the experiment of tossing two tetrahedra. We would expect that Px, y is positive since when X is large, Y tends to be large too. We calculated 4[XY], 4[X], and 4[ Y] in Example 18 and obtained 4[XY] = \365, 4[X] = 4-, and 4[ Y] = i ~. Thus cov [X, YJ = _\365 - t i ~ = t%. Now 4[X2] = 34 and 4[ y2] = \76; hence var [X] = t and var [y] = ~~. So,
Px,
y

J~Jll
4 64

t~

.'

1111

EXAMPLE 22 Find Px, y for X and Y iflx, y(x, y) = (x + y)I(o, 1)(x)I(o, 1)(Y)' We saw that 4[XY] = ~ and 4[X] = 4[ y] = 1\ in Example 19.. Now 4[X2] = 4[ y2] = 152; hence var [X] = var [y] = 1\14< Finally
1. _ J:.2...

Px, y =

144 11

1
= - -

T44

II

Does a negative correlation coefficient seem right?

1111

EXPECTATION

157

4.3

Conditional Expectations

In the following chapters we shall have occasion to find the expected value of random variables in conditional distributions, or the expected value of one random variable given the value of another. Conditional expectation Let (X, Y) be a two-dimensional random variable and g( . , . ), a function of two variables. The conditional expectation of g(X, Y) given X = x, denoted by C[g(X, Y)I X = x], is defined to be
C[g(X, Y) I X = xl =

Definition 21

f~oo g(x, Y)/YlxCYl x) dy

(23)

if (X, Y) are jointly continuous, and

C[g(X, Y) I X

= x] = L

g(x, Y)/Ylx(yil x)

(24)

if (X, Y) are jointly discrete, where the summation is over all. possible
~~~y

In particular, if g(x, y)=y, we have defined C[YIX=x]=C[Ylx]. C[Ylx] and C[g(X, Y)lx] are functions of x. Note that this definition can be generalized to more than two dimensions. For example, let (Xl' ... , X k , Y l , ... , Ym ) be a (k + m)-dimensional continuous random variable with density
/XI, ... , Xk , YI, ... , Ym(Xl' . " , Xk'

Yl' ., ., Ym); then

C[g(Xb ... , X k , Yb

... ,

Ym)IXl, ... , Xk]

... ,

.=

foo ... foo

-00 -00

g(x b

Xk'

Yl' ... , Ym)

EXAMPLE 23 In the experiment of tossing two tetrahedra with X, the number on the first, and Y, the larger of the two numbers, we found that

! /Ylx(yI2) = J. ! {
4

=2 for Y = 3
for Y for Y = 4

in Example 9.
_ 11

Hence C[ YI X

T'

= 2] = LY/YI x(y I X = 2) = 21- + 31 + 4 t . . . - .-. /f //

158

JOINT AND CONDmONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

EXAMPLE 24 For/x, y(x, y) = (x + y)I(o.l)(x)I(o, lly), we found that

/Ylx(Y Ix) = x+y
X

"2

I(o.I)(Y)

for 0 < x < I in Example 12. Hence

8[ Y I X = x]

X+ y d I y x+!
I

Y=

x+! 2

(X 1) - +3

for 0 < x < I.

//11

As we stated above, 8[g( Y) Ix] is, in general, a function of x, Let us denote it by hex); that is, hex) = 8[g(Y)lx]. Now we can evaluate the expectation of heX), a function of X, and will have 8[h(X)] = 8[8[g(Y)1 Xl]. This gi ves us
8[8[g(Y) IX]] = 8[h(X)]

= =
=

J:
00

h(x)fx(x) dx 8[g(Y)lx]fx(x) dx x) dY]fx(X) dx

oo

Jroo [J~ooU(Y)/Y1X(YI
oo

= Joo
-00

Joo
-00

g(Y)/Ylx(Ylx)/x(x)dydx

J~oo J~oog(Y)fx.Y(X' y) dy dx

= 8[g(Y)].
Thus we have proved for jointly continuous random variables X and Y (the proof for X and Y jointly discrete is similar) the following simple yet very useful theorem.

Theorem 6 Let (X, Y) be a two-dimensional random variable; then

8[g( Y)] = 8[8[g( Y) IX]],

(25)

and in particular
8[ Y] = 8[8[ YI X]]. (26)

IIII
Definition 22 Regression curve 8[ YI X = x] is called the regression IIII curve of Yon x. It is also denoted by ,LlYIX=x=P.Ylx

EXPECTATION

159

Definition 23 Conditional variance The variance of Y given X defined by var [YI X = x] = <9'[ y21 X = x] - (<9'[ YI X = X])2. Theorem 7
PROOF

x is

IIII

var [Y] = <9'[var

[YI X]] + var [<9'[ YI X]].

- <9'[(<9'[ YI X])2]

<9'[var [YI X]]

= <9'[ <9'[ y21 X]]

= <9'[ y2] - (<9'[ Y])2 - <9'[(<9'[ YI X])2] + (<9'[ y])2

= var [ Y] - <9'[(<9'[ YI X])2]

+ (<9'[<9'[ YI X]])2
IIII

= var [Y] - var [<9'[ YI X]].

Let us note in words what the two theorems say. Equation (26) states that the mean of Y is the mean or expectation of the conditional mean of Y, and Theorem 7 states that the variance of Y is the mean or expectation of the conditional variance of Y, plus the variance of the conditional mean of Y. We will conclude this subsection with one further theorem. The proof can be routinely obtained from Definition 21 and is left as an exercise. Also, the theorem can be generalized to more than two dimensions. Theorem 8 Let (X, Y) be a two-dimensional random variable and gl ( .) and g2(') functions of one variable. Then
(i) <9'[gl( y) + g2( Y) I X = x] = <9'[gl (Y) I X = x] + <9'[g2( Y) I X = x]. (ii) <9'[g1( y)g2(X) I X = x] = g2(X)<9'[gl( Y) I X = x].

IIII

4.4

Joint Moment Generating Function and Moments

We will use our definition of the expectation of a function of several variables to define joint moments and the joint moment generating function. Definition 24 Joint moments The joint raw moments of Xl' ... , X k are defined by <9'[X1 X~2 ... X~k], where the r/s are 0 or any positive integer; the joint moments about the means are defined by <9'[(Xl - /lxJ'l ... (Xk - /lxJ'k]. IIII
1

Remark If ri = rj = 1 and all other rm's are 0, then that particular joint moment ~bout the means becomes <9'[(Xi - /lx,)(Xj - /lx)], which is just the covanance between XI and Xi' IIII

160

JOINT AND CONDmONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

Definition 25 Joint moment generating function The joint moment generating function of (Xl, ... , X k) is defined by

mx" ... ,x.(t., ... , I.) = of

[expJ.r; xl

(27)

if the expectation exists for all values of t l , ... , tk such that -h < tj < h for some h > O,j = 1, ... , k. IIII The rth moment of Xj may be obtained from m Xt ... , Xk(tl, . , t k ) by differentiating it r times with respect to tj and then taking the limit as all the t's approach 0. Also 8[X~ Xj] can be obtained by differentiating the joint moment generating function r times with respect to ti and s times with respect to tj and then taking the limit as all the t's approach O. Similarly other joint raw moments can be generated. Remark mx(td = mx. y(tl' 0) = limmx. y(tl' t 2), andm y(t2) = mx. yeO, t 2)
t2-+0

= limmx
tl-+0

y(tl' t 2 ); that is, the marginal moment generating functions can

be obtained from the joint moment generating function.

IIII

An example of a joint moment generating function will appear in Sec. 5 of this chapter. 4.5 Independence and Expectation

We have already defined independence and expectation; in this section we will relate the two concepts. Theorem 9 If X and Yare independent and gl (.) and g2(') are two functions, each of a single argument, then
8[gl(X)g2( Y)]
PROOF

= 8[gl(X)]' 8[g2( Y)].

We will give the proof for jointly continuous random

variables.
t9'[gI(X)g2(Y)] = fOO fOO gl(X)g2(y)fx,y(X, y) dx dy
-00 -00

fOO fOO gl(X)g2(y)fx(x)fy(y) dx dy

-00 -00

= f~oogl(X)fx(X) dx . f~oog2(Y)fY(Y) dy = 8[g leX)] . t9'[g2(Y)]'

IIII

EXPECTATION

161

Corollary If X and Yare independent, then cov [X, Y] = O.

PROOF

Take gl(X) = x - Jlx and g2(Y) = Y - Jly; by Theorem 9,

cov [X, Y] = <9'[(X - Jlx)( Y - Jly)] = <9'[gl(X)g2( Y)]

= <9'[gl(X)]<9'[g2(Y)]
= <9'[X - Jlx] .

<9'[ Y - Jly]

= 0

since

~[X - Jlx] =

o.

IIII

Definition 26 Uncorrelated random variables Random variables X and Yare defined to be uncorrelated if and only if cov [X, Y] = o. IIII Remark The converse of the above corollary is not always true; that is, cov [X, Y] = 0 does not always imply that X and Yare independent, as the following example shows. IIII

EXAM PLE 25 Let U be a random variable which is uniformly distributed over the interval (0, 1). Define X = sin 2nU and Y = cos 2nU. X and Yare clearly not independent since if a value of X is known, then U is one of two values, and so Y is also one of two values; hence the conditional distribution of Y is not the same as the marginal distribution. ~[ Y] = Sb cos 2nu du = 0, and ~[X] = sin 2nu du = 0; so cov [X, Y]= ~[XY] = S6 sin 2nu cos 2nu du = 1- S6 sin 4nu du = o. IIII

Theorem 10 Two jointly distributed random variables X and Yare independent if and only if mx, y(tb t 2 ) = m X(tl)my(t 2) for all t1 , t2 for which -h < tj < h, i = I, 2, for some h > O. [Recall that mX(tl) is the moment generating function of X. Also note that mX(tl) = mx, y(tb 0).] X and Y independent imply that the joint moment generating function factors into the product of the marginal moment generating functions by Theorem 9 by taking gl (x) = et1x and g2(Y) = e2Y The proof in the other direction will be omitted.
PROOF

IIII
Remark Both Theorems 9 and 10 can be generalized from two random variables to k random variables. IIII

162

JOINT,&.ND CONDmONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

4.6

Cauchy-Schwarz Inequality
Theorem 11 Cauchy-Schwarz inequality Let X and Y have finite second moments; then (4[XY])2 = 14[XY] 12 < 4[X2)4[ y2), with equality if and only if P[ Y = cX] = I for some constant c. The existence of expectations 4[X), 4[ Y), and 4[XY) follows from the existence of expectations 4[X2) and 4[ y2). Define o <h(t) = 8[(tX - Y)2) = 4[X2)t 2 - 24[XY]t + 4[ y2). Now h(t} is a quadratic function in t which is greater than or equal to o. If h(t} > 0, then the roots of h(t) are not real; so 4(4[XY]}2 - 44[X 2)4[ y2) < 0, or (8[XY])2 < 8[X2)4[ y2). If h(t} = 0 for some t, say to, then 8[(t o X - Y)2] = 0, which implies P[to X = Y] = 1. IIII
PROOF

Corollary IPx, yl < 1, with equality if and only if one random variable is a linear function of the other with probability 1.
PROOF

Rewrite the Cauchy-Schwarz inequality as

14[UV)1 <

J 4[U2)4[ V 2), and set U = X 5

Jlx and V

Y-

Jly.

IIII

BIVARIATE NORMAL DISTRIBUTION

One of the important multivariate densities is the multivariate normal density, which is a generalization of the normal distribution for a unidimensional random variable. In this section we shall discuss a special case, the case of the bivariate normal. In our discussion we will include the joint density, marginal densities, conditional densities, conditional means and variances, covariance, and the moment generating function. This section, then, will give an example of many of the concepts defined in the preceding sections of this chapter.

5.1

Density Function Definition 27 Bivariate normal distribution Let the two-dimensional random variable (X, Y) have the joint probability density function
fx,y (x, y) =f(x, y)= 2 1 J 1taXay 1 2

xexp ( -

1 2 [(X - JlX) 2(1 - P ) ax

2p

X - Jlx Y- Jly + (Y - Jly) 2])

ax ay ay

(28)

BIVARIATE NORMAL DISTRIBUTION

163

z
z = f(x, y) for z > k

FIGURE 7

for - 00 < x < 00, - 00 < y < 00, where u y , u x , llx, lly, and p are constants such that -I < p < I, 0 < Uy , 0 < Ux, -00 < llx < 00, and - 00 < lly < 00. Then the random variable (X, Y) is defined to have a bivariate normal distribution. fill The density in Eq. (28) may be represented by a bell-shaped surface z = I(x, y) as in Fig. 7. Any plane parallel to the xy plane which cuts the surface will intersect it in an elliptic curve, while any plane perpendicular to the xy plane will cut the surface in a curve of the normal form. The probability that a point (X, Y) will lie in any region R of the xy plane is obtained by integrating the density over that region;
P[(X, Y) is in R]

= III(x, y) dy dx.
R

(29)

The density might, for example, represent the distribution of hits on a vertical target, where x and y represent the horizontal and vertical deviations from the central lines. And in fact the distribution closely approximates the distribution of this as well as many other bivariate populations encountered in practice. We must first show that the function actually represents a density by showing that its integral over the whole plane is I; that is,

I I
00 -00

00

f(x, y) dy dx

-00

= 1.

(30)

The density is, of course, positive.

To simplify the integral, we shall substitute

x-ll
U

and

y - lly v=--Uy

(31)

164

JOINT AND CONDITIONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

so that it becomes

I I
00 - 00

oo

- 00

2n

1 e -U-/(l_p2)](U2- 2pUV+V2) d v d u. 1 _ p2

.-

On completing the square on u in the exponent, we have

and if we substitute
u - pv w = --;-.===p2

JI 1

and

dw

du

JI- p2
2

the integral may be written as the product of two simple integrals

oo

- 00 "

-e -w /2 d w I J 1 e -v /2 d v, 2n 2n
2

oo

(32)

00

both of which are I, as we have seen in studying the univariate normal distribution. Equation (30) is thus verified. Remark The cumulative bivariate normal distribution

may be reduced to a form involving only the parameter p by making the fill substitution in Eq. (31).

5.2

Moment Generating Function and Moments

To obtain the moments of X and Y, we shall find their joint moment generating function, which is given by
.

rnX,y(tl'

t 2 ) = m(tl' t 2 )

= C[elX+t2Y] =

I I
00 -00

00

1X

t2

Yf(x, y) dy dx.

-00

Theorem 12 The moment generating function of the bivariate normal distribution is

BIVARIATE NORMAL DISTRIBUTION

165

PROOF

Let us again substitute for x and y in terms of u and v to

obtain
m(tl, t2)

The combined exponents in the integrand may be written

2(1

~ p2) [u 2 -

2puv

+ v2 -

2(1

p2)t,a x u - 2(1- p2)t 2 a y vI,

and on completing the square first on u and then on v, we find this expression becomes
- 2(1

~ p2) {[u -

2 + (1 - p2)(V - pt,ax - t2ay)2 pv - (I - p2)t,ax J -(1-p2)(tfui +2pt1t2UXUy +tiu~)},

which, if we substitute and becomes

_!w 2 -lz2 + !(trui + 2pt1t2UXUy + tiui),

and the integral in Eq. (34) may be written
m(th t 2) = e'lf.l x+ t1 J4y
x
=

exp[!(trui + 2Ptlt2 UXUy + t~u:)J

1 e-wl/2-z1/2 dw dz 2n

00

foo
- 00

-00

exp[t1Jlx

+ t2 Jly + !(tiui + 2Ptl t2 UXUy + ti ui)]

II1I

since the double integral is equal to unity.

Theorem 13 If (X, Y) has bivariate normal distribution, then

8[X]

= /lx,

8[ Y] = /ly,

var [X] = var [Y] =

u:,

ui,

cov [X, Y] = pux uy ,

166

JOINI' AND CONDmONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

and
Px , y = p.

The moments may be obtained by evaluating the appropriate derivative of m(tl' t 2) at tl = 0, t2 = O. Thus,
PROOF

2 2 =..J1x + Gx .

Hence the variance of X is

Similarly, on differentiating with respect to t 2 , one finds the mean and variance of Y to be Jly and G~. We can also obtain joint moments

by differentiating m{t., t 2 ) r times with respect to tl and s times with respect to t2 and then putting tl and t2 equal to O. The covariance of X and Y is
4[(X - Jlx)(Y ~ Jly)]

= 8[XY = 8[XY] -

XJly - YJlx Jlx Jly

+ JlxJly]

82 a 8 m(t., t 2 )
t1 t2

tl = t2 = 0

Jlx Jly

pGxGy .

Hence, the parameter p is the correlation coefficient of X and Y.

III1

Theorem 14 If (X, Y) has a bivariate normal distribution, then X and Y are independent if and only if X and Yare uncorrelated.
X and Yare uncorrelated if and only if cov [X, Y] = 0 or, equivalently, if and only if Px, Y = P = O. It can be observed that if p = 0, the joint density f(x, y) becomes the product of two univariate
PROOF

normal distributions; so that p = 0 implies X and Yare independent. We know that, in general, independence of X and Y implies that X and Y are uncorrelated. 1I1I

BIVARIATE NORMAL DISI1UBUTION

167

5.3

Marginal and Conditional Densities

Theorem 15 If (X, Y) has a bivariate normal distri bution, then the marginal distributions of X and Yare univariate normal distributions; tbat is, X is normally distributed with mean JJ.x and variance ai, and Y is normally distributed with mean JJ.y and variance ai. The marginal density of one of the variables X, for example, is by definition
PROOF

/x(x)

QO

f(x, y) dy;
y
~

-QO

and again substituting

V=--

JJ.y

ay

and completing the square on v, one finds that

00

/x(x)=f
-QO

2nax

J1 1(X -

1 x exp [ - 2

JJ.X)2 1 2 (V ax 2(1 - p )

P X - JJ.X)2] dv. ax

Then the substitutions and show at once that

fx(X) =

~i exp [ _ ~ (X

;;x) 2],
ay

the univariate normal density. be found to be

fy(y)

Similarly the marginal density of Y may

==

,,2nay

1 exp[1 - - (y - p,y) 2] .
2

1111

Theorem 16 If (X, Y) has a bivariate normal distribution , then the conditional distribution of X given Y = y is normal with mean JJ.x + (pax/ay)(y ~ JJ.y) and variance ai(I - p2). Also, the conditional distribution of Y given X = x is normal with mean JJ.y + (payjax)(x - JJ.x) and variance ai(l _ p2).

168

JOINT AND CONDmONAL DISTRIBUTIONS, STOCHASTIC INDEPENDENCE

IV

The conditional distributions are obtained from the joint and marginal distributions. Thus, the conditional density of X for fixed values of Y is
PROOF

f(x, y) fXly(xly) = fy(y) ,

and, after substituting, the expression may be put in the form

!XIY(x Iy)

J21!UX~1 _ pi exp {-

2uj(/- p2)

[x - I1x

-7.X

(y -

I1r)]'},

(35)

which is a univariate normal density with mean f1x + (paxlay)(y - f1y) and with variance ai(1 - p2). The conditional distribution of Y may be obtained by interchanging x and y throughout Eq. (35) to get
!Ylx(Y Ix)

= J21!

p urJi _ p2 ex { - 2ui(11_ p2) [Y -

I1r - p::

(x - I1X)]

(36)

IIII
As we already noted, the mean value of a random variable in a conditional distribution is called a regression curve when regarded as a function of the fixed variable in the conditional distribution. Thus the regression for X on Y = Y in Eq. (35) is f1x + (paxlay)(y - f1y), which is a linear function of y in the present case. For bivariate distributions in general, the mean of X in the conditional density of X given Y = Y will be some function of y, say g(.), and the equation
x =g(y)

when plotted in the xy plane gives the regression curve for x. It is simply a curve which gives the location of the mean of X for various values of Y in the conditional density of X given Y = y. For the bivariate normal distribution, the regression curve is the straight line obtained by plotting
x = f1x

+ ~ (y ay

pax

f1y),

as shown in Fig. 8. The conditional density of X given Y = y, fXIY(x Iy), is also plotted in Fig. 8 for two particular values Yo and Yl of Y.

PROBLEMS

169

FIGURE 8

PROBLEMS
1

3 4

Prove or disprove: (a) If P[X> Y] = 1, then S[X] > SlY]. (b) If S[X] > SlY], then P[X> y] = 1. (e) If S[X] > S[ Y], then P[X> y] > O. Prove or disprove: (a) If Fx(z) > Fy(z) for alJ z, then SlY] > S[X]. (b) If S[y] > C[X], then Fx(z) > Fy(z) for an z. (e) If Cry] > S[X], then Fx(z) > Fy(z) for some z. (d) If Fx(z) Fy(z) for all z, then P[X = Y] = 1. (e) If Fx(z) > F y(z) for all z, then P{X < Y] > O. (I) If Y = Xl I, then Fx(z) Fy(z + 1) for all z. If X I and X 2 are independent random variables with distribution given by P[X, 1] =P[X~ = 1] ! for i = 1, 2, then are Xl and X I X 2 independent? A penny and dime are tossed. Let X denote the number of heads up_ Then the penny is tossed again. Let Y denote the number of heads up on the dime (from the first toss) and the penny from the second toss. (a) Find the conditional distribution of Y given X = I. (b) Find the covariance of X and Y. If X and Y have joint distribution given by

lx,
(a)

y(x, y)

U(O,y)(x)I(o. J)(Y).

Find COy [X, Y]. (b) Find the conditional distribution of Y given X = x. Consider a sample of size 2 drawn without replacement from an urn containing three ba1ls, numbered 1, 2, and 3. Let X be the number on the first ball drawn and Y the Jarger of the two numbers drawn. (a) Find the joint discrete density function of X and Y. (b) Find P[X = 11 Y 3]. (e) Find cov [X. Yl.

170

JOINT AND CONDITIONAL DISTRIBUTIONS. STOCHAS'l1C INDEPENDENCE

IV

Consider two random variables X and Y having a joint probability density function

Ix. r(X, y) =

lxyl(o.Jr)(y)I(o. 2)(X).

(a) Find the marginal distributions of X and Y. (b) Are X and Y independent? If X has a Bemoul1i distribution with parameter p (that is, p[X = 1] = p = 1 ~ p[X = OD, 8[ YI X = 0] = 1, and 8[YI X = 1] = 2, what is 8[ Y]?

9 Consider a sample of size 2 drawn without replacement from an urn containing three balls, numbered 1, 2, and 3. Let X be the smaller of the two numbers drawn and Y the larger. (a) Find the joint discrete density function of X and Y. (b) Find the conditional distribution of Y given X = 1. (c) Find cov [X, Y]. 10 Let X and Y be independent random variables, each having the same geometric distribution. Find P[X = Y]. 11 If F() is a cumulative distribution function: (a) Is F(x, y) = F(x) + F(y) a joint cumulative distribution function "1 (b) Is F(x, y) = F(x)F(y) a joint cumu1ative distrIbution function? (c) Is F(x, y) = max [F(x), F(y)] a joint cumulative distribution function? (d) Is F(x, y) = min [F(x), F(y)] a joint cumulative distribution function "1 12 Prove
F,,(x) + Fr(y) ~ I <F". y{x, y)

<VF,,(x)Fr(Y)

for al1 x, y.

13 Three farr coins are tossed. Let X denote the number of heads on the first two coins, and let Y denote the number of tails on the last two coins. (a) Find the joint distribution of X and Y. (b) Find the conditional distribution of Y given that X = 1. (c) Find cov [X, Y]. 14 Let random variable X have a density function 1(), cumulative distribution function F( .), mean p, and variance (12. Define Y = or; + fJx, where or; and fJ are constants satisfying ~ 00 < or; < 00 and {J > O. (a) Select or; and fJ so that Y has mean 0 and variance 1. (b) What is the correlation coefficient between X and Y? (c) Find the cumulative distrIbution function of Y in terms of or:. (J, and F( .). (d) If X is symmetrically distrIbuted about p, is Y necessarily symmetrically distributed about its mean "1 (HINT: Z is symmetrically distributed about constant C if Z ~ C and ~(Z ~ C) have the same distribution.) 15 Suppose that random variable X is uniformly distributed over the interval (0, 1); that is, I,,(x) = 1(0. 1)(x). Assume that the conditional distribution of Y given X = x has a binomia1 distribution with parameters n and p = x; i.e.,
p[Y = yl X
=

x] =

C)

x'(1 - x),,-'

for y = 0, 1, . , n.

PROBLEMS

171

Find "[y]. Find the distribution of Y. 16 Suppose that the joint probabiJity density function of (X, Y) is given by
(a) (b)

Ix. y(x, y) =

[1 - IX(l - 2x)(l - 2y)]I(o.1)(x)I(o. 1)(Y)'

where the parameter IX satisfies -1 < IX < 1. (a) Prove or disprove: X and Yare independent if and only if X and Yare uncorrelated.

An isosceles triangle is formed as indicated in the sketch. (b) If (X. Y) has the joint density given above, pick IX to maximize the expected area of the triangle. (c) What is the probability that the triangle falls within the unit square with corners at (0, 0), (1, 0), (1, 1), and (0, 1) 7 *(d) Find the expected length of the perimeter of the triangle. 17 Consider tossing two tetrahedra with sides numbered 1 to 4. Let Y 1 denote the smalJer of the two downturned numbers and Y 2 the larger. (a) Find the joint density function of Y1 and Y 2 (b) Find pry! ~ 2, Y 2 2]. (e) Find the mean and variance of Y 1 and Y 2 (d) Find the conditional distribution of Y 2 given Y 1 for each of the possible values of Y 1 (e) Find the correlation coefficient of Y1 and Y 2 18 Let/x. rex, y) = e-(x+7)I (0. <X!)(x)I(o. <X!)(y) (a) Find P[X> lJ. (b) Find p[1 < X + Y < 2]. (e) Find P[X < YI X < 2 Yl. (d) Find m such that p[X + Y < m] = i. (e) Find p[0 < X < 11 Y = 2]. (f) Find the correlation coefficient of X and Y. *19 Let /x. rex, y) = e -"(1 - e-x)l(o. "J(x)l[O, <X!)(y) + e-x(l - e-")I(o. x)(y)I[o. <X!)(x). (a) Show that/x. y(', .) is a probat?ility density function. (b) Find the marginal distributions of X and Y. (e) Find S[YI X = x] for 0 < x. (d) Find P[X 2, Y 2]. (e) Find the correlation coefficient of X and Y. (n Find another joint probability density function having the same marginals.

172

JoINT AND CONDITIONAL DISTRIBUTIONS, STOCH~C INDEPENDENCE

IV

Suppose X and Y are independent and identically distributed random variables with probability density function f(} that is symmetrical about o. (a) Prove that Pfl X + YI 21 XI] > 1_ (b) Select some symmetrical probability density function f('}, and evaluate P[IX YI <2IXI] *21 Prove or disprove: If 8[YI X] = X,8[XI Y] = Y, and both 8[X2] and 8[y2] are finite, then P[X = Y] = 1. (Possible HINT: P[X = Y] = 1 if var [X Y] = 0.) 22 A multivariate Chebyshev inequaJity: Let (Xl, .. _, Xm) be jointly distributed with
"'20

8 [Xl]

ILl and var [Xl]

m

= u~ for j = 1, "', m. Define Al = {I Xl

r2.

ILl I

< Vmtul}'

Show that P[ () A l ] > 1 1=1

for t>O.

23

24

25

26

27

Let fx( .) be a probability density function with corresponding cumulative distribution function F x(). In terms of fx(') and/or F x('}: (a) Find p[X> XO + .::lxl X>xo]. (b) Find P[xo < X <Xo + .::lxl X> xo]. (c) Find the limit of the above divided by .::lx as .::lx goes to O. (d) Evaluate the quantities in parts (a) to (c) for fx(x} = 'Ae-;,xI(o. OC)(x). Let N equal the number of times a certain device may be used before it breaks. The probability is p that it will break on anyone try given that it did not break on any of the previous tries. (a) Express this in terms of conditional probabiJities. (b) Express it in terms of a density function, and find the density function. Player A tosses a coin with sides numbered 1 and 2. B spins a spinner evenly graduated from 0 to 3. B's spinner is fair, but A's coin is not; it comes up 1 with a probability p, not necessarily equal to 1. The payoff X of this game is the difference in their numbers (A's number minus B's). Find the cumulative distribution function of X. An urn contains four balls; two of the balls are numbered with aI, and the other two are numbered with a 2. Two bal1s are drawn from the um without replacement. Let X denote the smaller of the numbers on the drawn balls and Y the larger. (a) Find the joint density of X and Y. (b) Find the marginal distribution of Y. (c) Find the cov [X, Y]. The joint probability density function of X and Y is given by
fx. rex, y}

= 3(x + y)I(o. 1)(x

y)l(o. l)(x}I(o. 1)(y}.

(Note the symmetry in x and y.) (a) Find the marginal density of X. (b) Find P[X + Y < .5]. (c) Find 8[ YI X = x]. (d) Find cov [X, Y].

PROBLEMS

173

28 The discrete density of X is given by Ix(x) = xl3 for x binomia1 with parameters x and !; that is,

1, 2, and IYI x(Ylx) is

I" x(Ylx} ~P[Y~YI X

~xl ~ (;) U).

29

30

31
32

33

34

35

36

for Y = 0, ... , x and x = 1, 2. (a) Find 8[X] and var [X). (b) Find 8[ Y). (c) Find the joint distribution of X and Y. Let the joint density function of X and Y be given by Ix. y(x, y) = 8xy for 0 < x < y < 1 and be 0 e1sewhere. (a) Find 8[ YI X = x). (b) Find 8[XYIX=x]. (c) Find var [YI X = x). Let Y be a random variab1e having a Poisson distribution with parameter "-. Assume that the conditiona1 distribution of X given Y = y is binomia11y distributed with parameters y and p. Find the distribution of X, if X = 0 when Y = O. Assume that X and Yare independent random variab1es and X ( y) has binomia1 distribution with parameters 3 and t (2 and i). Find P[X = Y). Let X and Y have bivariate norma1 distribution with parameters p'x = 5, p. Y = 10, ai = 1, and a~ = 25. (a) If p > 0, find p when P[4 < Y < 161 X = 5] = .954. *(b) If p = 0, find P[X + Y < 16). Two dice are cast 10 times. Let X be the number of times no Is appear, and 1et Y be the number of times two Is appear. (a) What is the probabiJity that X and Y wi11 each be 1ess than 3? (b) What is the probabiHty that X + Y wi11 be 4? Three coins are tossed n times. (a) Find the joint density of X, the number of times no heads appear; Y, the number of times one head appears; and Z, the number of times two heads appear. (b) Find the conditiona1 density of X and Z given Y. Six cards are drawn without rep1acement from an ordinary deck. (a) Find the joint density of the number of aces X and the number of kings Y. (b) Find the conditiona1 density of X given Y. Let the two-dimensiona1 random variab1e (X, Y) have the joint density
lx, y(x, y)
(a) (c)

= 1(6 -

,x - y)I(O, 2)(x)I(2.4ly).

(b) Find8[Y2IX=x]. Find 8[YI X = x], Find var [YI X = x], (d) Show that 8[ y] = 8[8[ YI X]]. (e) Find 8[XYI X = x). 37 The trinomia1 distribution (muhinomia1 'with k + 1 = 3) of two random variab1es X and Y is given by Ix. y(x, y)

x!y!(n - x - y)!

n' '

pXqY(1

-/J _

q)"-X-Y

for x, y = 0, 1, ... , n and x

+ y < n, where 0 <p, 0 .-:::;: q, and p + q < 1.

174

JOINT AND CONDmONAL DISTRIBUTIONS, STOCHASTIC iNDEPENDENCE

IV

Find the marginal distribution of Y. (b) Find the conditional distribution of X given Y, and obtain its expected value. (e) Find p[X, Y]. 38 Let (X, Y) have probability density function/x. y(x, y), and let u(X) and v(Y) be functions of X and Y, respectively. Show that
(a)

,.

8[u(X)v(Y) I X

x]

u(x)8[v(Y) I X

x].

39 If X and Yare two random variables and 8[YI X = x] = ft, where ft does not depend on x, show that var [Y] = 8[var [YI Xl], 40 If X and Yare two independent random variables, does 8[ YI X = x] depend
onx?

41

If the joint moment generating function of (X, Y) is given by mx. y(II' ( 2 ) exp[l(lf+ In] what is the distribution of Y?

42 Define the moment generating function of YI X = x. Does my(l) = 8[mYI x(I)]? 43 Toss three coins. Let X denote the number of heads on the first two and Y denote the number of heads on the last two. (a) Find the joint distribution of X and Y. (b) Find 8[YI X = 1]. (e) Find px. y. (d) Give a joint distribution that is not the joint distribution given in part (a) yet has the same marginal distributions as the joint distribution given in part (a). 44 Suppose that X and Y are jointly continuous random variables, /Ylx(ylx) = I(x. X+I)(Y), and/x(x) = 1(0. O(x). (a) Find 8[Y]. (b) Find cov [X, Y]. (e) Find P[X + Y <]]. (d) Find/xI y(xly) 45 Let (X, Y) have a joint discrete density function

Ix.

y(x, y) pf(1 - PI)I-Xp~(l - P2)I-Y[l

+ (x -PI)(Y -P2)]/(0.1)(x)/(0. l)(Y),

where 0 <PI < 1, 0 <P2 < 1, and -1 ~ < 1. Prove or disprove: X and Y are independent if and only if they are uncorrelated. *46 Let (X, Y) be jointly discrete random variables such that each X and Y have at most two mass points. Prove or disprove: X and Yare independent if and only if they are uncorrelated.

v
DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

1 INTRODUCTION AND SUMMARY

As the title of this chapter indicates, we are interested in finding the distributions of functions of random variables. More precisely, for given random variables, say Xl' X 2 , . , X n , and given functions of the n given random variables, say gl(, .... ), g2(', ... , .), ... , gle(-, ... , .), we want, in general, to find the joint distribution of Y 1, Y 2 , ... , Y Ie , where Yj = giX1, .. , Xn),j = 1,2, ... , k. If the joint density of the random variables Xl' X 2 , , Xn is given, then theoretically at least, we can find the joint distribution of Ylo Y2 , , Yle . This follows since the joint cumulative distribution function of Y1 , , Yle satisfies the following:
FYI, ... , Yk(Ylo ... , YIe) = P[ Y1 < Y1; .. ; Yk < YIe]

= P[gl(X1"", XII) < Yl; ;gle(X1,., Xn)<YIe]

for fixed Y1' , Yle' which is the probability of an event described in terms of Xl' .. , X n and theoretically such a probability can be determined by integrating or summing the joint density over the region corresponding to the event. The problem is that in general one cannot easily evaluate the desired probability

176

DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

for each Yl' ... , Yk' One of the important problems of statistical inference, the estimation of parameters, provides us with an example of a problem in which it is useful to be able to find the distribution of a function of joint random variables. In this chapter three techniques for finding the distribution offunctions of random variables will be presented. These three techniques are called 0) the cumulative-distribution-function technique, alluded to above and discussed in Sec. 3, (ii) the moment-generating-function technique, considered in Sec. 4, and (iii) the transformation technique, considered in Secs. 5 and 6. A number of important examples are given, including the distribution of sums of independent random variables (in Subsec. 4.2) and the distribution of the minimum and maximum (in Subsec. 3.2). Presentation of other important derived distributions is deferred until later chapters. For instance, the distributions of chisquare, Student's t, and F, all derived from sampling from a normal distribution, are given in Sec. 4 of the next chapter. Preceding the presentation of the techniques for finding the distribution of functions of random variables is a discussion, given in Sec, 2, of expectations of functions of random variables. As one might suspect, an expectation, for example, the mean or the variance, of a function of given random variables can sometimes be expressed in tenns of expectations of the given random variables. If such is the case and one is only interested in certain expectations, then it is not necessary to solve the problem of finding the distribution of the function of the given random variables. One important function of given random variables is their sum, and in Subsec. 2.2 the mean and variance of a sum of given random variables are derived, We have remarked several times in past chapters that our intermediate objective was the understanding of distribution theory. This chapter provides us with a presentation of distribution theory at a level that is deemed adequate for the understanding of the statistical concepts that are given in the remainder of this book.

EXPECTATIONS OF FUNCTIONS OF RANDOM VARIABLES

2.1

Expectation Two Ways

An expectation of a function of a set of random variables can be obtained two different ways. To illustrate, consider a function of just one random variable, say X. Let g(') be the function, and set Y = g( X). Since Y is a random

EXPECT ATIONS OF FUNCTIONS OF RANDOM VARlABLES

177

variable, 8[ Y] is defined (if it exists), and 8[g( X)] is defined (if it exists). For instance, if X and Y = g( X) are continuous random variables, then by definition
00

8[Y] = f_ooyfy(y) dy,

(I)

and
8[g(X)] =

f~oog(X)fx(X) dx;

(2)

but Y = g(X), so it seems reasonable that 8[ Y] = 8[g(X)]. This can, in fact, be proved; although we will not bother to do it. Thus we have two ways of calculating the expectation of Y = g(X); one is to average Y with respect to the density of Y, and the other is to average g(X) with respect to the density of X. In general, for given random variables X h .. , X,., let Y = g(Xl' . , XII); then 8[ Y] = 8[g(Xh .. , X,.)], where (for jointly continuous random variables)
8[ Y] = foo yfy(y) dy
-00

(3)

and

8 [g(Xl , ... , XII)]

oo
.

-00

.. 00
g(x 1 ,, x lI )fxl ..... XJXl
, ... ,

XII) dXl ... dx,..

-00

(4)

In practice, one would naturally select that method which makes the calculations easier. One might suspect that Eq. (3) gives the better method of the two since it involves only a single integral whereas Eq. (4) involves a multiple integral. On the other hand, Eq. (3) involves the density of Y, a density that may have to be obtained before integration can proceed.

EXAMPLE 1 Let X be a standard normal random variable, and Ietg(x) = x 2 For Y =g(X) = X 2 ,
00

8[Y]

yfy(y) dy,
-00

and
8[g(X)] = 8[X2] =

00

x:tx(x) dx.

-00

Now

178

DISTRIBtmONS OF FUNCTIONS OF RANDOM VARIABLES

and

using the fact that Y has a gamma distribution with parameters r A. = t. (See Example 2 in Subsec. 3. I below.)

! and JI II

2.2 Sums of Random Variables

A simple, yet important, function of several random variables is their sum.

Theorem 1 For random variables Xl' ... , X"

.c[~ x,] = ~ .c[X,),

and var[f
1

(5)

Xl]

=f
1

var[Xi ]

+ 2 L L cov[X
i<j

j ,

Xj]'

(6)

PROOF

That.c [ ~ X, =

.c [X,] follows from a property of expec-

tation (see the last Remark in Subsec. 4.1 of Chap. IV).

var[

~ x,] =.c[ (~x,- e[~ x.]) 2J =.c[ (~(XI - .c[X,])n

=
=

eLt, J.(X,- .c[XaXX

i=l j;.l

j -

.c[X

j ])]

" L " 8[(Xj L

8[Xi ))(Xj

8[XJ)]

i=l

" var[Xi) + 2 I I L
i<j

COV[Xi' Xj]'

JIJI

Corollary

If Xl' ... , X" are uncorrelated random variables, then

var

[~ XI] = ~ var[X.J.

/1/1

The following theorem gives a result that is somewhat related to the above theorem inasmuch as its proof, which is left as an exercise, is similar.

EXPEcrATIONS OF FUNCTIONS OF RANDOM VARIABLES

179

Theorem 2 Let Xl' ... , XII and Y1, , Ym be two sets of random variables, and let al, , all and b1, , bm be two sets of constants; then

Corollary If Xl, ... , X II are random variables and al' ... , a" are cons tants, then
(8)
=

L af var[X,] + L L a,aj cov[X

i=l
i:f:.j

II

j ,

X j ].
II

In particular, if Xl' ... , XII are independent and identically distributed random variables with mean 11 x and variance then
8[XII ] = Ilx,

0'; and if X II = (I/n) LXi'

1

and

var [XII] = -.
n

O'x

(9)

Let m theorem; then

PROOF

n, Y, = Xi' and b i

= ai' i = I, ... , n in the above

var[~ aiXi] = COy [t a,X" ~ b

and Eq. (8) follows from Eq. (7). from Eq. (8), set a, = I/n and is routinely derived as

Yl
/11/

0';

To obtain the variance part of Eq. (9) = var [Xd. The mean part of Eq. (9)

Corollary If Xl and X 2 are two random variables, then

var [Xl

+ X 2 ] = var [Xd + var [X 2 ] + 2 cov [XI'

X 2 ].

(10)

JIll
Equation (10) gives the variance of the sum or the difference of two random variables. Clearly

180

DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

2.3

Product and Quotient

In the above subsection the mean and variance of the sum and difference of two random variables were obtained. It was found that the mean and variance of the sum or difference of random variables X and Y could be expressed in terms of the means, variances, and covariance of X and Y. We consider now the problem of finding the first two moments of the product and quotient of X and Y.

Theorem 3 Let X and Y be two random variables for which var [X YJ

exists; then
G[X YJ =
P,x p,y

+ cov [X, YJ,

(12)

and var [X YJ
= p,~

+ p,i var [YJ + 2P,x p,y cov [X, YJ - (cov [X, YJ? + G[(X - p,X)2(Y - .uy}2] + 2.u yG[(X - P,x)2 (Y - p,y)] + 2P,x G[(X var [X]
PROOF

(13)
P,x)( Y - p,y)2].

XY =

P,x p,y

+ (X -

P,x)p,y

+ (Y -

p'y}p'x

+ (X

- P,x)( Y - p'y).

Calculate G[X YJ and G[(Xy)2] to get the desired results.

/11/

Corollary If X and Yare independent, G[X YJ = P,x P,y, and var [X YJ =

p'~ var [X]

+ p,i var [YJ + var [X] var [Y].

If X and Yare independent,
P,x)2( Y - p,y)2]

PROOF

G[(X

= G[(X =

.ux)2]G[( Y - p,y)2]

var [Xl var [Y],

G[(X - p,X)2( Y - p,y)] = G[(X - p,X)2]G[Y - p,y] = 0,

and

Note that the mean of the product can be expressed in terms of the means and covariance of X and Y but the variance of the product requires higher-order moments. In general, there are no simple exact formulas for the mean and variance of the quotient of two random variables in terms of moments of the two random variables; however, there are approximate formulas which are sometimes useful.

CUMULATIVE-DISTRIBUTION-FUNCTION TECHNIQUE

181

Theorem 4
X] 8' [ -

RP,x ::--

Ily

2 Ily

1 cov[X, Yl+3"var Jlx [YJ ,

Ily

(14)

and var PROOF

[X] Y

R::

~x)2(var[Xl
y

Jlx

_ 2cov[X, + var[Yl 2
p,y

Yl)

Jlx Jly

(15)

To find the approximate formula for 8'[XI y], consider the Taylor series expansion of xly expanded about (Ilx, Jly); drop all terms of order higher than 2, and then take the expectation of both sides. The approximate formula for var [XI Y] is similarly obtained by expanding in a Taylor series and retaining only second-order terms. IIII Two comments are in order: First, it is not unusual that the mean and variance of the quotient XI Y do not exist even though the moments of X and Y do exist. (See Examples 5, 23, and 24.) Second, the method of proof of Theorem 4 can be used to find approximate formulas for the mean and variance of functions of X and Yother than the quotient. For example,
1 var[X] ;z g(x, y) 8'[g(X, Y)] R:: g{J1x, Jly) + -2
I

vx

I
PX,py

+ - var[Y]
2

vy

a2
2

g(x, y)
px.Py

+ cov[X,

Y] ~y ;lX g(x, y) v v

a2

,
px.py

(16)

and var[g(X, Y)]

R::

var[X](: g(x, y)
X
PX,py

}2 + var[Y](~ g(x, y) oy
. ~ g(x, y)
Px.py

)2
PX,py

+ 2 cov[X, Yl(:x g(x, Yli

ay

).
px.py

(17)

3 CUMULATIVE-DISTRIBUTION-FUNCTION TECHNIQUE

3.1

Description of Technique

If the joint distribution of random variables X h . . . , XII is given, then, theoreticaJIy, the joint distribution of random variables of Yt , ... , Yk can be determined, where Yj=giXt, ... , XII),j= 1, ... , k for given functionsgt(', .. " ,), ... ,

182

DISTRIBUTIONS OF FUNcn:ONS OF RANDOM VARIABLES

By definition, the joint cumulative distribution function of Y1 , . , . , Yk is FYi' ... , Yk(Yl' ... , Yk) = P[ Y1 Yl; ... ; Yk < yd. But for each Yl' "" Yk the event {Y1 <Yl; ... ; Yk <Yk} {Ul(X1 , ... , X,,) <Yl; ... ; Uk(Xl , . , X,,) < Yk}' This latter event is an event described in terms of the given functions Ul ( " ... , .), ... , gk(', .. , .) and the given random variables Xl' ... , X". Since the joint distribution of Xl' ... , X" is assumed given, presumably the probability of event {Ul (Xl' ... , X,,) Yl;' .. ; Uk( Xl, ... , X,,) < Yk} can be calculated and consequently Fyl. ... , Yk(', ... , .) determined. The above described technique for deriving the joint distribution of Yl , ... , Yk will be called the cumulative-distrihution-/unction technique. An important special case arises if k I; then there is only one function, say g( Xl' ... , X,,), of the given random variables for which One needs to derive the distribution.
gk( " .,., . ).

EXAMPLE 2 Let there be only one given random variable, say X, which has a standard normal distribution. Suppose the distribution of Y = g(X) = X2 is desired.
FI'(Y)

= pry <

y] = P[X 2 < y]

P[

-JY ~ X <JY] = tP(Jy) - tP( -Jy)

";y ";y = 2 t/J(u) du = 2 o 0

tu2 Jedu 2n

J2n

.2 f)/ 1
0

2Jz e-

tz

dz =

f)/ 1
0

1 tz r(!-) J2z e- dz, for y > 0,

which can be recognized as the cumulative distribution function of a gamma distribution with parameters r = ! and A. = !. //// Other applications of the cumulative-distribution-funct~on technique expounded above are given in the following three subsections. i,

3.2

Distribution of Minimum and Maximum

Let Xl' ... , X" be n given random variables. Define Yl = min [Xl' ... , X,,] and Y" = max [Xh .,., X,,]. To be certain to understand the meaning of Yn = max [Xl' .,., X,,], recall that each Xi. is a function with domain n, the sample space of a random experiment. For each 00 E n, Xl(oo) is some real number. Now Y" is to be a random variable; that is, for each 00, YnCoo) is to be

CUMULATIVE-DISTRIBUTION-FUNCTION TECHNIQUE

183

some real number. As defined, Y,,(w) = max [X1(w), ... , X,,(w)]; that is, for a given w, Y,,(w) is the largest of the real numbers Xl (w), ... , X,,(w). The distributions of YI and Y" are desired. F y" (y) = P[ Yn y] = P[X I < Y; .. ; X" < y] since the largest of the X,'s is less than or equal to y if and only if all the X,'s are less than or equal to y. Now, if the X;'s are assumed independent, then
P[X I <y; ... ; Xn<Y]= [lP[Xt<y]= [lFxlY);
i= 1 I
I
" n

so the distribution of Yn = max [X I ' ... , Xn] can be expressed in terms of the marginal distributions of Xl' ... , X n If in addition it is assumed that all the Xl' ... , X" have the same cumulative distribution, say F x('), then

" FX,(y) = [Fx(y)]n. [I

1= 1

We have proved Theorem 5.

,Theorem 5 ...

If X I' ... , Xn are independent random variables and Yn = max [Xl' ... , X,,], then
Fy"(Y)

" Fx.(Y) = [I
1= I

(18)

If Xl' ... , Xn are independent and identically distributed with common cumulative distribution function Fx( .), then Fy"(y)
=

[Fx(Y)]".

(19)

1I11
Corollary If X., ... , X" are independent identically distributed continuous random variables with common probability density function/xC .) and cumulative distribution function Fx( .), then

PROOF

Similarly,
P[YI~y]=1

P[Y1

> y] = I - P[XI > y;

,.

. .. ,

Xn> y]

184

DISTRlBUfiONS OF FUNCTIONS OF RANDOM VARIABLES

since. YI is greater than y if and only if every Xi > y. independent, then 1 - P[X I > Y; .. ; X" > y]

And if Xl, ... , X" are

= 1 - " P[X i > Y] = 1 - " [1 - F xlY)].

i= I

i= 1

If further it is assumed that Xl, .. , X" are identically distributed with common cumulative distribution function ..F x ( .), then
I -

1= I

" [] - Fxj(Y)] = J n

[J - Fx(Y)]",

and we have proved Theorem 6.

Theorem 6 If Xl, ... , X" are independent random variables and Y1

min [Xl' ... , X,,], then

. (2])

And if X I, .. , X" are independent and identically distributed with common cumulative distribution function Fx( .), then Fy1(Y)
1 - [I - Fx(Y)]".

(22)

IIII
Corollary If Xl, . , X" are independent identically distributed continuous random variables with common probability density ix(') and cumulative distribution F x ('), then

PROOF

d dy

-Fy1 (Y) =

n[1

II II

EXAMPLE 3 Suppose that the life of a: certain light bulb is exponentially distributed with mean 100 hours. If 10 such light bulbs are installed simultaneously, what is the distribution of the life of the light bulb that fails first, and what is its expected life? Let X i denote the life of the ith light bulb; then Y 1 = min [Xl' . , X IO ] is the life of the light bulb that fails first. Assume that the X i'S are independent.

CUMULATiVE-DISTRIBUTION-FUNCTION TECHNIQUE

t 85

Now fx'(x) = 1 ~Oe-.tfoXI(O. oo)(x), and

Fx,(x)
(1
e-1~oX)I(o, oo)(X);

so

!Yl(Y)

= JO(e-l~o")10-1(ntoe-1~o")I(o.

oolY)

/o~-looo)ll(o. (0)(Y),

which is an exponential distribution with parameter A. = -10; hence 8[Y1 ] = 1/,1. = 10. 1111
3.3

Distribution of Sum and Difference of Two Random Variables Theorem 7 Let X and Y be jointly distributed continuous random variables with density fx. y(x, y), and let Z = X + Y and V = X Y. .Then,
00

Iz(z) = f-

oo

Ix. y(x, z - x) dx =

f_ Ix. y(z 00 00 -00

y, y) dy,

(24)

and
00

Iy(v) = fix. y(x, x

-00

v) dx =

foo Ix. y(v + y, y) dy.

(25)

We will prove only the first part of Eq. (24); the others are proved in an analogous manner.
PROOF

Fz(z)

= P[Z < z] = P[X -f:. Y

z]

II Ix. y(x, y) dx dy

d dx = ( , [f!x,y(X, u- x) dU] dx
=
y) y ]

C, [()x, y(x,

by making the substitution y Now

u-

x.

!iZ)=d;;Z) = ~ {(o [f!x,y(X,U-X)dX] dU}

=

foo Ix. y(x, z -00

x) dx.

1/1/

186

DISTRIBUTIONS OF FUNCTIONS OF RANPOM VARIABLES

Coronary If X and Yare independent continuous random variables and Z = X + Y, then

fz1.z) =fx+y(z) =
PROOF

J_!Y(Z
00

x)fx(x) dx =

00

fx(z - y)fy(y) dy.

(26)

~-oo

Equation (26) follows immediately from independence and Eq. (24); however, we will give a direct proof using a conditional distribution formula. [See Eq. (II) of Chap. IV.]
P[Z <z] =P[X + Y < z]

00

P[X + Y <zlX = x]fx(x)dx z]fx(x) dx

-00

= J P[x + Y
-00

00

J
dz

,00 -00

Fy(z

x)fx(x) dx.

Hence,
fz(z)

= dF z(z) =

-d [J"'OO Fy(z

dz

x)fx(x) dx ]

00

= J"'OO dF y(z - x) fx(x) dx 00 dz

=

J~!Y(Z - x)fx(x) dx.

//1/

Remark The formula given in Eq. (26) is often called the convolution formula. In mathematical analysis, the function fz(') is caned the convolution of the functions fy( .) and f x( . ). IIII EXAMPLE 4 Suppose that X and Yare independent and identically distributed with density fx(x) = fy(x) = 1(0, 1)(x). Note that since both X and Yassume values between 0 and I, Z = X + Y assumes values between o and 2.
fz(z)

J~!y(z -

x)fx(x) dx

= f~oo 1(0, l)(Z - x)/(o, l)(x) dx

foo
-00

{I (0, z)( x)/(O, 1)(z) + I (z-l, 1)(x)I[1, 2)(Z)} dx

/(0, 1)(Z)

Jo dx + 1[1, 2)(Z) {-1 dx

+ (2 Z)/[l, 2)(Z).

:z;

= zl(o, l)(Z)

1I11

CUMULATlVE-DISTRIBUTlON-FUNCTlON TECHNIQUE

187

FIGURE 1

3.4 Distribution of Product and Quotient Theorem 8 Let X and Y be jointly distributed continuous random variables with density Ix, y(x, y), and let Z = XY and U = xl Y; then

iz(z) =
and

roo Th Ix. Y(

x,

~) dx = roo ~ Ix. y(~, y) dy,

lu(u) =

(27)

00

Iyl/x. y(UY, y) dy.

(28)

-00

Again, only the first part of Eq. (27) will be proved. Fig. I for z > 0.)
PROOF

(See

Fz(z) = P[Z < z] =

f:

xy:s;Z
00

II Ix. y(x, y) dx dy
0 00

[fOOfx. y(x, y) d Y] dx + foo [IZ,xlx. y(x, y) d Y] dx,

x/x

188

DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

which on making the substitution u

= xy
Z

= fO
-00

[J-oo/x,y(x,~) dU] dx + roo [f fx.Y(x,~) dU] dx

Z

x x

"0

-00

x x

(ro [(, IJx,y(x,~) dX] du + (ro [( ~fx,+,~) dX] du = L. [fro I~I fx,y(x,~) dX] du;

hence

liz) = dFz(z) dz
=

C, ~ fx,.( x,~) dx.

x

IIII

EXAMPLE 5 Suppose X and Yare independent random variables, each uniformly distributed over the interval (0, I). Let Z = X Y and U = X I Y.

Iz(z) = roo _111 Ix. y(x, ~) dx

"'-00

lrAu)

00

- 00

IY IIx. y(uy, y) dy
(see Fig. 2)

f~ Iy I{/(O, o(u)1 (o.l)(Y) + 1[1, oolu)/(o. '/llly)} dy

00

= 1(0, l)(u)
1 =2 1(0,

fo y dy + 1[1. OO)(u) f 1(1)2 l)(u) + 2 ;; 1[1,oo)(u).

1

1{1l

y dy

Note that rS[XI Y] = 8[U] = 1 J~ u du ferent from 8[X]18[ Y] = I.

+ 1- J~(llu) du = 00, quite difIIII

MOMENT-GENERATING-FUNCTION TECHNIQUE

189

~=--_U

FIGURE 2

4 4.1

MOMENT-GENERATING-FUNCTION TECHNIQUE Description of Technique

There is another method of determining the distribution of functions of random variables which we shall find to be particularly useful in certain instances. This method is built around the concept of the moment generating function and will be called the moment-generating-function technique. The statement of the problem remains the same. For given random variables X I ' .. , X,. with given density fXI' ... , x,,(Xt, ... , X,.) and given functions gd " ... , .), ... , gk( . , .. " .), find the joint distribution of Y1 = g1 (X h ... , X,.), ... , Yk = gk(X" .. , X,.), Now the joint moment generating function of Y 1 , , Y k , if it exists, is t) = @[elYl++tkYk] m Yl .... Yk(t... l' 'k

Ix!, ... , X,,(X t , . , X,.) n dXi'

n

(29)

i= 1

If after the integration of Eq. (29) is performed, the resulting function of t 1 , , tk can be recognized as the joint moment generating function of some known joint distribution, it will follow that Y1, ''', Yk has that joint distribution by virtue of the fact that a moment generating function, when it exists, is unique and uniquely determines its distribution function. For k > 1, this method will be of limited use to us because we can recognize only a few joint moment generating functions. For k = 1, the moment generating function is a function of a single argument, and we should have a better chance of recognizing the resulting moment generating function.

190

DISTRIBUTIONS OF FUNCITONS OF RANDOM VARIABLES

This method is quite powerful in connection with certain techniques of advanced mathematics (the theory of transforms) which, in many instances, enable one to determine the distribution associated with the derived moment generating function. The most useful application of the moment-generating-function technique will be given in Subsec. 4.2. There it will be used to find the distribution of sums of independent random variables.

EXAMPLE 6 Suppose X has a normal distribution with mean 0 and variance I. Let Y = X 2 , and find the distribution of Y.

for

t<

2.'

which we recognize as the moment generating function of a gamma with parameters r = t and ,t = t. (It is also called a chi-square distribution 1/// with one degree of freedom. See Subsec. 4.3 of Chap. VI.)

EXAMPLE 7 Let Xl and X 2 be two independent standard normal random variables. Let Y1 = Ul(X 1 , X 2) = Xl + X 2 and Y2 = U2(X h X 2 ) = X 2 - Xl' Find the joint distribution of Y1 and Y 2
m

Yl.Y2

(t l' t) = 2

(0

.!.P[e Y 1t1 + Y2t2]

1

= ~"'[e(X + X2)tt + (X2 - x d 2] = 8[eX1(t1 -(2)+X2(t1 +t2)]

f

= cC[eXt(ft-t2)]cC[eX2(tl +( = mXl(t 1 - t 2)mxi t l + t 2) (11 - t2)2 (11 + t2)2

2 )]

exp

exp

exp(t; + t~)

2ti 2ti exp 2 exp 2:

= mY1(tl)mY2(t2)'

MOMENT-GENERATING-FUNCTION TECHNIQUE

191

We note that YI and Y2 are independent random variables (by Theorem 10 of Chap. IV) and each has a normal distribution with mean 0 and variance 2. ., II1I In the above example we were able to manipulate expectations and avoid performing an integration to find the desired joint moment generating function. In the following example the integration will have to be performed.

EXAMPLE 8 Let Xl and X 2 be two independent standard normal random variables. Let Y = (X 2 - X I )2/2, and find the distribution of Y.
my(/) = of[exp YI]

= of [exp (X 2 - ; X I )2 t]
1
TC

OO

foo
-00

-00

-2 exp

[(X2 - XI)2 2 tI)

xi + X~] 2 dXI dX2

+ 2X,X2 I + x~(1
-

=
=

roo roo 2~ exp{- ~ [x:(1 foo ~ exp [- ~ x~(1- t)]

- 00

t)]} dx, dX2

2TC

{f_
OO
-00

OO

00

exp - -2- Xl

[1 - t (2 +

2XtX2 1_ t

t)] dXI }dX

t

[X~(1 exp ~2TC 2

~

t)] exp

X2 2 2(1 - t)
---....:2::...-_

1 x J1 _ t 1 = J1 _ I

{roo JJ~ exp [ roo exp [ - ~ (1 -

_1 2 I (x,
I - 1,2

t2/y]

dx,) dX2

}i] dX2

- Ji - t Jl - 21 . J 1 - t J2n

J 1-

Jt=U

foo
-00

( 1 1 - 2t ) exp -:2 1 _ t x~ dX2

= (1- 2/)-! =

(! ! I)!' for t < 1/2,

which is the moment generating function of a gamma distribution with parameters r = ! and A. = ~.; hence,

192

DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

4.2 Distribution of Sums of Independent Random Variables

In this subsection we employ the moment-generating-function technique to find the distribution of the sum of independent random variables.

Theorem 9 If X 1 X are independent random variables and the moment generating function of each exists for all -h < t < h for some ,. h > 0, let Y = I Xi; then
II

my(t)

= t9'[ exp LXi t] = 0

j=

,.

mx.(t)

for

-h<t<h.

PROOF

=0
using Theorem 9 of Chap. IV.

II

G[ell

1= l

=0
j=

II

mxlt)
1

1/1/

The power and utility of Theorem 9 becomes apparent if we recall Theorem 7 of Chap. II~ which says that a moment generating function, when it exists, determines the distribution function. Thus, if we can recognize

0
i"" 1

mxlt) as

the moment generating function corresponding to a particular distribution, then we have found the distribution of I Xi'
1

,.

In the following examples, we

will be able to do just that.

EXAMPLE 9 Suppose that Xl, ... , X II are independent Bernoulli random variables; that is, P[X i = I] = p, and P[X f = 0] = I - p. Now
mxlt) = pe

+ q ..
(pe'

So

mr xlt) =

.0 mxlt) =
1=1

II

+ q)lI,

the moment generating function of a binomial random variable; hence

n

LXi has a binomial distribution with parameters nand p.

1

lIlt

MOMENT-GENERATING-FUNCTION TECHNIQUE

193

EXAMPLE 10 Suppose that Xl, ... , XII are independent Poisson distributed random variables, Xi having parameter A. i Then
mx.(t)
=

C[e tXi ] = exp Ale' - 1),

and , hence
mr. xlt) =
1=1

nmxlt) nexp Ai(e

II II

1) = exp

i=l

L Ai(e

1),

which is again the moment generating function of a Poisson distributed random variable having param,eter L Ai . So the distribution of a sum of independent Poisson distributed random variables is again a Poisson distributed random variable with a parameter equal to the sum of the I111 individual parameters.

EXAMPLE 11 Assume that Xl, ... , XII are independent and identically distributed exponential random variables; then

So
mI:x,(t) =

,n
II

mxlt) =

A )11 A. - t '

which is the moment generating function of a gamma distribution with parameters n and A; hence,

f I: x.() X -

An
r(n)
X

n-l-lxj

() (0, (0) X ,

the density of a gamma distribution with parameters n and A..

1II1

EXAMPLE 12 Assume that Xl' ... , Xn are independent random variables and then and

194

DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

Hence
m'f,o;x.(t) =
i=l

nmOtxlt)

exp[(I aif,J.i)t

+ t(I afuf)t 2] ,

which is the moment generating function of a normal random variable; so

The above says that any linear combination (that is, L ai Xi) of independent normal random variables is itself a normally distributed random variable. (Actually, any linear combination of jointly normally distributed random variables is normally distributed. Independence is not required.) In particular, if

x '" N(px, oJ),

and X and Yare independent, then
X + Y", N(Jlx + Jly, uk + u~),

and

x-

Y '" N(p.x - Jly, u; + u~).

If Xl' ... , Xn are independent and identically distributed random variables distributed N(Jl, ( 2 ), then

X.=!I: X

I -

(I'. :2);
/1/1

that is!, the sample mean has a (not approximate) normal distribution.

In the above examples we found the exact distribution of the sums of certain independent random variables. Other examples, including the important result that the sum of independent identically distributed geometric random variables has a negative binomial distribution, are given in the Problems. One
n

is often more interested in the average, that is, (lIn) LXi' than in the sum.
1

Note, however, that if the distribution of the sum is known, then the distribution of the average is readily derivable since

F(1Jn)'X,(z)

p[;; I: X, < z] = P[L XI < nz] = Fr,x,(nz).

(30)

In Examples 9 to 12 above, where we derived the distribution of a sum, we have in essence also derived the distribution of the corresponding average. One of

MOMENT-GENERATING-FUNCTION TECHNIQUE

195

the most important theorems of all probability theory, thecentral-limittheorem, gives an approximate distribution of an average. We will state this theorem next and then again in our discussion of sampling in Chap. VI, where we will outline its proof. Theorem 10 Central-limit theorem If for each positive integer n, Xl, ... , X,. are independent and identically distributed random variables with mean Jlx and variance ui-, then for each z
FzJz) converges to CI>(z) as n approaches
00,

(31)

where

We have made use ofEq. (9), which stated that ~[XII] = Jlx and var [XII] = uiln. Equation (31) states that for each fixed argument z the value of the cumulative distribution function of ZII' for n = I, 2, ... , converges to the value CI>(z). [Recall that CI>(.) is the cumulative distribution function of the standard normal distribution.] Note what the central-limit theorem says: If you have independent random variables X I, ... , XII , ... , each with the same distribution which has a mean and variance, then XII = (lIn) LXi" standardized" by subtracting its mean and then dividing by its standard deviation has a distribution that approaches a standard normal distribution. The key thing to note is that it does not make any difference what common distribution the Xl' ... , XII' ... have, as long as they have a mean and variance. A number of useful approximations can be garnered from the central~limit theorem, and they are listed as a corollary. Corollary If Xl' ... , XII are independent and identically distributed random variables with common mean Jlx and variance ui, then

[a < X. u /
x

-;X
n

< b]

~ <I>(b) -

CI>(a) ,

(32)
(33)

P[~ < XII < d] ~ CI> (~- Jl~) uxlJn

or

CI>(C - Jlx) ,

ux/Jn

IIII

196

DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

Equations (32) to (34) give approximate values for the probabilities of certain events described in terms of averages or sums. The practical utility of the central-limit theorem is inherent in these approximations. At this stage we can conveniently discuss and contrast two terms that are a vital part of a statistician's vocabulary. These two terms are limiting distribution and asymptotic distribution. A distribution is called a limiting distribution function if it is the limit distribution function of a sequence of distribution functions. Equation (31) provides us with an example; <J>(z) is the limiting distribution function of the sequence of distribution functions F zt ('), Fzi'), ... , F zn{'), . Also <J>(z) is called the limiting distribution of the sequence of random variables Z1, Z 2, .. , Z,., .. . . On the other hand, an asymptotic distribution of a random variable, say Y,., in a sequence of random variables Y 1 , Y 2 , Y,., ... is any distribution that is approximately equal to the actual distribution of Y,. for large n. As an example [see Eq. (33)], we say that X,. has an asymptotic distribution that is a normal distribution with mean J-lx and variance ui/n. Note that an asymptotic distribution may depend on n whereas a limiting distribution does not (for a limiting distribution the dependence on n was removed in taking the limit). Yet the two terms are closely related since it was precisely the fact that the sequence ZI' Z2, ... , Z,., ... had limiting standard normal distribution that allowed us to say that X,. had an asymptotic normal distribution with mean J.lx and variance ui/n. The idea is that if the distribution of Z,. is converging to <J>(z), then for large n the distribution of Z,. must be approximately distributed

N(O, I). But if Z,. = (X,. ~ J-lx)/(ux/Jn) is approximately distributed N(O, I), then X,. is approximately distributed N(;tx, ui/n). In concluding this section we give two further examples concerning sums. The first shows how expressing one random variable as a sum of other simpler random variables is often a useful ploy. The second shows how the distribution of a sum can be obtained even though the number of terms in the sum is also a random variable, something that occasionally occurs in practice.

EXAMPLE 13 Consider n repeated independent trials, each of which has possible outcomes {) 1, , Ok + l ' Let Pj denote the probability of outcome o. on a particular trial, and let X denote the number of the n trials resulting J J in outcome {}j' j = 1, ... , k + l. We saw that (XH ... , X k ) had a multinomial distribution. N ow let if ath trial results in outcome otherwise;
{}j

MO~NT-GENERATING-FUNCTION TECHNIQUE

197

then Xj =

Zjl%'

Now suppose we want to find coY [Xi' Xj ],

In-

a=l

tuitively, we might suspect that such covariance is negative since when one of the random variables is large another tends to be small.

by Theorem 2. Now if IX i" p, then ZiP and Zja are independent since they correspond to different trials, which are independent. Hence

p=la=l

I I

,.

COY [Zip, Z ja] =

a=l

COY [Zil%' Z jl%]'

But COY [Zia, Zja] = C[ZiaZja] C[Zia]C[Zja], and C[ZiI%Zja] = 0 since at least one of Zia and Zja must be O. Now C[Zia] = Pi' and C[Zja] = Pj;
socov[Xi,Xj]=-nPiPj'

IIII

EXAMPLE 14 Let Xl' ... , Xn , ... be a sequence of independent and identically distributed random variables with mean Jlx and variance uj. Let N be an integer-valued random variable, and define SN =

L Xi; that is, SN

i= 1

is the sum of the first N X/s, where N is a random variable as are the X/so Thus SN is a sum of a random number of random variables. Let us assume that N is independent of the X/so Then C[SN] = C[C[SNIN]] by Eq. (26) of Chap. IV. But C[SNI N = n] = C[Xl + ... + Xn] = nJlx; so 8[SNIN] =NJlx, and C[C[SNI N]] = C[NJlx] = JlxC[N] = JlNJlX' Similarly, using Theorem 7 of Chap. IV, var [SN] = C[var [SN IN]]

+ var [C[SN IN]]

= C[Nui] + var [NJlx] = uiC[N] + Jli var [N] = JlN . ui + u~ . Jli.

Suppose now that N has a geometric distribution [see Eq. (14) of Chap. III] with parameter p, Xi has an exponential distribution with parameter 1, and we are interested in the distribution of S N Further assume independence of N and the X/so Now, for z> 0,
P[SN < z]

n=1

L P[SN < zlN = n]P[N = n]

ao

198

DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

(by using the fact that a sum of independent and identically distributed exponential random variables has a gamma distribution)

f ).pez

lU e(l- p )lll

du

= ).p

f ez

lpu

du

=1-

e- lpz

That is, SN has an exponential distribution with parameter pl. Recall that [see Eq. (14) of Chap. ITI] $[N] = lip and var [N] = (I p)lp2; also, $[X] = II)., and var [Xl = IJ).2. SO, as a check of the formulas for the mean and variance derived above, note that
1 1 IlN Ilx = - . -

1
p).

p ).

and
2 22

IlN(JX

+ (JNllx =

p).2 + 7

11

1-p1
).2

= (p).) 2 ,

which are the mean and variance, respectively, of an exponential distribution with parameter pl. JIJI

5 THE TRANSFORMATION Y=g(X)

The last of our three techniques for finding the distribution of functions of given random variables is the transformation technique. It is discussed in this section for the special case of finding the distribution of a function of a unidimensional random variable. That is, for a given random variable X we seek the distribution of Y = g(X) for some function g( . ). Discussion of the general case is deferred until Sec. 6 below. Both the notation Y = g(X) and the notation y = g(x) will appear in the ensuing paragraphs; y = g(x) is the usual notation for the function or transformation specified by g('), and Y = g(X) defines the random variable Y as the function g(') of the random variable X. 5.1

Distribution of Y = g(X)

A random variable X may be transformed by some function g(.) to define a new random variable Y. The density of Y, fy(y), will be determined by the transformation g(.) together with the density fx(x) of X.

THE TRANSFORMATION

g(X)

199

First, if X is a discrete random variable with mass points Xl' X 2, , . then the distribution of Y = g(X) is determined directly by the laws of probability. If X takes on the values Xl' X 2 ' with probabilities/x(xl ),/x(x2)' ... , then the possible values of Yare determined by substituting the successive values of X in g('). It may be that several values of X give rise to the same value of y, The probability that Y takes on a given value, say Yj' is
/y(y) =

L
(i!g(xt)= YJ)

fx(Xi)'

(35)

EXAMPLE 15 Suppose X takes on the values 0, I, 2, 3, 4, 5 with probabilities/x(0),/x(l),/x(2),/x(3),/x(4), and/x(5). If Y = g(X) = (X - 2)2, note that Y can take on values 0, I, 4, and 9; then /y(O) = /x(2), /y(I) = /x(l) + /x(3),/y(4) = /x(O) + /x(4) , and/y(9) = fx(5). IIII Second, if X is a continuous random variable, then the cumulative distribution function of Y = g(X) can be found by integrating/x(x) over the appropriate region; that is,
Fy(Y) = P[Y:::;;; y]
P[g(X) < y]

/x(x) dx.

(36)

{x:g(X)5)'}

This is just the cumulative-distribution-function technique.

EXAMPLE 16 Let X be a random variable with uniform distribution over the interval (0, 1) and let Y = g(X) = X 2 , The density of Y is desired. Now
Fy(Y) = pry < y]
=

2 P[X < y] =

fx(X) dx =

for

<

(x: x 2 s)'}

f )' dx = Jy
..;~

y < I; so

and therefore
I I
/Y(Y)=

2Jy I(o.1)(Y)'
1

IIII

Application of the cumulative-distribution-function technique to find the density of Y = g(X), as in the above example, produces the transformation technique, the result of which is given in the following theorem.

200

DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

Theorem 11 Suppose X is a continuous random variable with probability density function f x(). Set X = {x: f x(x) > O}. Assume that:

y = g(x) defines a one-to-one transformation of X onto 'D. (ii) The derivative of x = g-l(y) with respect to y is continuous and nonzero for y E 'D, where g-l(y) is the inverse function of g(x); that is, g-l(y) is that x for which g(x) = y.

0)

Then Y

= g(X) is a continuous random variable with density

The above is a standard theorem from calculus on the change of variable in a definite integral; so we wil1 only sketch the proof. Consider the case when I is an interval. Let us suppose that g(x) is a monotone increasing function over X; that is, g'(x) > 0, which is true if and only if (dldy)g-l(y) > 0 Qver 'D. For y E 'D, Fy(y) = P[g(X) < y] = P[X ~ g-l(y)] =FX(g-l(y)), and hence fy(y) = (dldy)Fy(y) = [(dldy)g-l(y)] fX(g-l(y by chain rule of differentiation. On the other hand, if g(x) is a monotone decreasing function over I, so that g'(x) < 0 and (dldy)g-l(y) < 0, then Fy(y) = P[g(X) < yl = P[X > g-l(y)] = I - Fx (g-l(y,andthereforefy(y) = - [(dldy)g-l(y)]fx(g-l(y = I(dldy)g-\y) I fX(g-l(y for y E 'D. IIII
PROOF

EXAMPLE 17

Suppose X has a beta distribution. What is the distribution of Y = -IogeX? I = {x:fx(x) > O} = {x:O < x < I}. y =g(x) = ~Iogex defines a one-to-one transformation of X onto 'D = {y: y > O}. x = g-l(y) = e-"', so (dldy)g-\y) = -e- Y, which is continuous and nonzero for y E ID. By Theorem II,
fy(y)

=
=

dyg-l(y) fX(g-l(yI!}(y)
Y

B(a, b) e

Y)Q -1 (I

_ e - Y\b - 11

(0, (0)

() Y

I - B(a, b) e- aY(I - e Y)b 11(0, OO)(y),

In particular, if b = I, then B(a, b) = I/a; so fy(y) = ae-ay/(O, oo)(Y), an exponential distribution with parameter a. IIII

THE TRANSFORMATION

Y=g(X)

201

EXAMPLE 18 Suppose X has the Pareto density/x(x) the distribution of Y = loge X is desired.
IY(Y)

= (}X- 9 - 1[[1. oo)(x) and

~ ~ g-l(y)

fx(g-l(yl'f)(Y)
(}e9y [[o. OO)(Y)

= eY(}(e'j-9-1 1[1. OO)(e Y ) =

IIII

The condition that g(x) be a one-to-one transformation of X onto ~ is unnecessarily restrictive. For the transformation y = g(x), each point in X will correspond to just one point in ~ ; but to a point in ~ there may correspond more than one point in X, which says that the transformation is not one-to-one, and consequently Theorem II is not directly applicable. If, however, X can be decomposed into a finite (or even countable) number of disjoint sets, say Xl, ... , Xm , so that y = g(x) defines a one-to-one transformation of each Xi into ~, then the joint density of Y = g(X) can be found. Let x = gi l(y) denote the inverse of y = g(x) for x E Xi. Then the density of Y = g(X) is given by

where the summation is over those values of i for which g(x) = y for some value of x in Xi.

EXAMPLE 19 Let X be a continuous random variable with density /x(), and let Y = g(X) = X2. Note that if X is an interval containing both negative and positive points, then y = g(x) = x 2 is not one-to-one. However, if X is decomposed into Xl = {x: x E X, X < O} and X2 = {x: x EX, X >O}, then y = g(x) d~nes a one-to-one transformation on each Xi. Note that g11(y) = y and g2 l(y) = By Eq. (37),

JY.

In particular, if
/x(x)

= H)e- 1x1 ,
= 2 JY e
1 I

then
lY(Y)
r

-'Or

Y[(o.ooly);

202

DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

or, if

then

IIII

5.2 Probability Integral Transform

If X is a random variable with cumulative distribution F x ('), then F x (') is a candidate for g(') in the transformation Y = g(X). The following theorem gives the distribution of Y = Fx(X) if F x (') is continuous. Since F x (') is a nondecreasing function, the inverse function Fi \.) may be defined for any value of y between and I as; Fi ley) is the smallest x satisfying Fx(x) :2::: y.

Theorem 12 If X is a random variable with continuous cumulative distribution function Fx(x), then U = FxCX) is uniformly distributed Over the interval (0, I). Conversely, if U is uniformly distributed over the interval (0, I), then X = Fi l(U) has cumulative distribution function Fx ().
PROOF

P[U

< u]

= P[Fx(X)

u for 0

< u < I. Conversely, P[X

u] = P[X < Fi 1 (u)] = Fx(F-x 1 (u = x] = P[Fil(U) < x] = P[U < Fx(x)]

= Fx(x).

IIII

In various statistical applications, particularly in simulation studies, it is often desired to generate values of some random variable X. To generate a value of a random variable X having continuous cumulative distribution function F x ('), it suffices to generate a value of a random variable U that is uniformly distributed over the interval (0, I). This fonows from Theorem 12 since if U is a random variable with a uniform distribution over the interval (0, I), then X = Fi 1 (U) is a random variable having distribution F x( . ). So to get a value, say x, of a random variable X, obtain a value, say u, of a random variable U, compute Fx l( u), and set it equal to x. A value u ofa random variable U is called a random number. Many computer-oriented random-number generators are available.

TRANSFORMATIONS

203

EXAMPLE 20 Fx(x) = (I - e- lx )l(o, oo)(x), Fx 1(y) = -(l/A.) loge (I - y); so -O/l) loge (l - U) is a random variable having distribution (1 - e-;'X) 1(0, OO)(X) if U is a random variable uniformly distributed over the interval (0, I). / // /

The transformation Y = Fx(X) is called the probability integral transformation. It plays an important role in the theory of distribution-free statistics and goodness-of-fit tests.

TRANSFORMATIONS

In Sec. 5 we considered the problem of obtaining the distribution of a function of a given random variable. It is natural to consider next the problem of obtaining the joint distribution of several random variables which are functions of a given set of random variables.

6.1

Discrete Random Variables

Suppose that the discrete density function fXh . . , x n (x 1 , , xn) of the ndimensional discrete random variable (Xl' ... , Xn) is given. Let l: denote the mass points of (Xl' ... , X n ); that is,

Suppose that the joint density of Yl = gl(Xl , ... , X n), .. , Yk = gk(Xl , ... , Xn) is desired. It can be observed that Yl , , Yk are jointly discrete and P[Yl =Yl; ... ; Yk = Yk] =!Yl, ... ,Yk(Yl'''',Yk) = I!xt .... ,xn(Xl , ... ,xn),where the summation is over those (Xh .. , xn) belonging to I for which (Yl, ... 'Yk) =
(gl(Xl' ... , XII)' . , gk(Xh ... , XII'

EXAMPLE 21 by

Let (Xl' X 2 , X 3 ) have a jOint discrete density function given

(0,0,0) (0,0,1) (0,1,1) (1,0,1) (1,1,0) (1,1,1)

204

DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

Find the joint density of Y1 = 91(X 1, X 2 , X 3) Y 2 = 92(X1, X 2 , X 3) = IX 3 - X21

= Xl + X 2 + X3 and

X = {(O, 0, 0), (0, 0, 1), (0, 1, 1), (1, 0, 1), (1, 1,0), (1, 1, I)}.
!Yt. Y 2(0, 0) =!X X2.X3(0, 0, 0) = !Y to Y2(1, 1) =!Xt, X 2. X 3(0, 0,1) = !Y to Y2(2, 1) =!x lo x2.xi 1, 0,1)

L i,

!Yt,Y2(2, 0) =!XI,X2.X3(0, 1,1) = ~,

+ !x x 2.x3(1, 1,0) = i,
IIII

and

6.2

Continuous Random Variables

Suppose now that we are given the JOInt probability density function fXI ... Xn (X1' ... , XII) of the n-dimensional continuous random variable (Xl' ... , XII)' Let

X = {(Xl' ... , XII) :fxt. ... Xn (X1, ... , XII) > O}.

(38)

Again assume that the joint density of the random variables Y1 = 91 (X 1, ... , XII)' ... , Yk = 9k( Xl' ... , XII) is desired, where k is some integer satisfying I <k < n. If k < n, we will introduce additional, new random variables Yk + 1 = 9k+1(X 1, ... , X,,), ... , Y II = 911 (X 1 , , XII) for judiciously selected functions 9k+ l' .. , 911; then we will find the joint distribution of Yh .. , YII , and finally we will find the desired marginal distribution of Y1 , , Yk from the joint distribution of Y1 , , YII . This use of possibly introducing additional random variables makes the transformation Y1 = 91(Xb ... , x,,), ... , YII = 911(X1, ... , XII) a transformation from an n-dimensional space to an n-dimensional space. Henceforth we will assume that we are seeking the joint distribution of Y1 = 91(X1, ... , XII)' ... , YII = 911(X1, ... , XII) (rather than the joint distribution of Y1, ... , Yk) when we have given the joint probability density of Xl, ... , XII' We will state our results first for n = 2 and later generalize to n > 2. LetfX1>XiX1' X2) be given. Set X ={(Xb x 2):fxl.xi x 1' X2) > O}. We want to find the joint distribution of Y1 = 91(X 1, X 2) and Y2 = 92(X b X 2) for known functions 91(', .) and 92(', '). Now suppose that Y1 = 91(X1' X2) and Y2 = 92(X1' X2) defines a one-to-one transformation which maps X onto, say, ~. Xl and X2 can be expressed in terms of Y1 and Y2; so we can write, say, Xl = 911(Y1' Y2) and X2 = 921(Yb Y2)' Note that X is a subset of the X1X2 plane and ~ is a subset of the Y1Y2 plane. The determinant

TRANSFORMATIONS

205

aXl aYl aX2 aYl

aXl aY2 aX2 aY2

(39)

will be called the Jacobian of the transformation and will be denoted by J. The above discussion permits us to state Theorem 13.

Theorem 13 Let Xl and X 2 be jointly continuous random variables with density function!xl~xixl' X2). Set X = {(Xh X2):!X1.Xz(x l , X2) > O}.
Assume that:
Yl = gl(Xlo X2) and Y2 = g2(Xl' x 2) defines a one-to-one transformation of X onto ~. (ii) The first partial derivatives of Xl = gll(Yh Y2) and X2 = g2"l(Yl' Y2) are continuous over ~. (iii) The Jacobian of the transformation is nonzero for (Yl' Y2) E ~.
(i)

Then the joint density of Yl = gl(X l , X 2) and Y2 = g2(X l , X 2) is given by

fYI, YZ(Yl' Y2)
PROOF

IJ Ifxt.xz(gll(Yb Y2)' g2"l(Yl' Y2IIl)(Yl' Y2)

(40)

We omit the proof; it is essentially the same as the derivation of the formulas for transforming variables in double integrals, which may be found in many advanced calculus textbooks. ~ is that subset of the YlY2 plane consisting of points (Yl, Y2) for which there exists a (Xl' X2) EX such that (Yl' Y2) = (gl(Xl, X2)' g2(Xl' X2. IIl)(Yl' >'2) = I~(gll(Yl' Y2), g2"l(Yl' Y2. IIII EXAMPLE 22 Suppose that Xl and X 2 are independent random variables, each uniformly distributed over the interval (0, 1). Then!xI,xz(xl, x 2 ) = I(o.1)(x l )I(o,1)(x2) X = {(Xl' X2): O<Xl < I and 0<X2 < I}. Let Yl = gl(Xh X2) = Xl + X2 and Y2 = g2(Xl , X2) = X2 - Xl; then Xl = 1 -!(Yl - Y2) = g1 (Yl' Y2)' and x 2 = t(Yl + Y2) = g2"l(Yh Y2).
-

aXl aYl aX2

aX - l aY2

-!

J=

=-

aYl

aX2

1 2

aY2

206"

DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

FIGURE 3

the boundary !(Yl + Y2) = 0 ofID, the boundary Xl = I of X goes into the boundary t(Yl - Y2) = I of ID, and the boundary X2 = I of X goes into the boundarY-!(Yl + Y2) = 10fID. Now the transformation is one-to-one, the first partial derivatives of g~ 1 and g2 1 are continuous, and the Jacobian is nonzero; so

X and ID are sketched in Fig. 3. Note that the boundary Xl = 0 of X goes into the boundary !(Yl - Y2) = 0 ofID, the boundary x 2 = 0 of X goes into

= !1(0, t) ( Yl
=

2 Y2) 1(0,1) (Yl + 2 Y2)

{~

for (Yl' Y2)EID otherwise.

IIII

EXAMPLE 23 Let Xl and X 2 be two independent standard normal random variables. Let Yl = Xl + X 2 and Y2 = X l lx 2 Then
-l( ) YIY2 Xl=91 Yl'Y2 =1+Y2

and

-l( Yl' Y2 ) = 1 Yl X2 = 92

+ Y2

J=

Y2 1 +Y2
1 1 + Y2

Yl (1 + Y2)2 Yl (1 + Y2)2

= -

Yl(Y2 + 1) Yl = (1 + Y2)2 (1 + Y2)3

TRANSFORMATIONS

207

To find the marginal distribution of, say, Y 2 , we must integrate out YI ; that is

!r (Y2) = f_!rt. r2(vl' Y2) dYl

2

00

- 2n (I

+ Y 2)2

foo
_
00

IY I I

ex [_

~ (1 + Y~)Yi]
2 (1

+ Y 2)2

Y 1-

Let

then
du
=

(I + Y~) (I + Y2) 2 Yl dYt

and so

a Cauchy density. That is, the ratio of two independent standard normal 1/1/ random variables has a Cauchy distribution.

EXAMPLE 24 Let Xi have a gamma density with parameters ni and A. for i = I, 2. Assume that Xl and X 2 are independent. Again, we seek the jOint distribution of Y1 = Xl + X 2 and Y2 = X l /X 2 -

Xl

= 91

-l(

) YIY2 Ylt Y2 = 1 + Y2

and

X2 = 92

-1(

Yl' Y2 = 1

+ Y2 '

Yl_

208

DISTRIBUTIONS OF FUNCrIONS OF RANDOM VARIABLES

hence

We see thatfYl. Y2(Yt, Y2) = f Yl(Yt)fY/Y2); so Yt and Y 2 are independent. Also, we see that the distribution of Yt = X. + X 2 is a gamma distribution with parameters nt + n z and A. If nl = nz = I, then Y2 is the ratio of two independent exponentially distributed random variables and has density

a density which has an infinite mean.

1/1/

EXAMPLE 25 Let Xi have a gamma distribution with parameters ni and A for i = l~ 2~ and assume X t and X 2 are independent. Suppose now that the distribution of Yt = X./(X t + X 2 ) is desired. We have only the one function Yt = gt(x., X2) = x.l(xt + X2); so we have to select the other to use the transformation technique. Since Xl and x 2 occur in the exponent of their joint density as their sum, x. + x 2 is a good choice. Let Y2 t : Xl + X2; then X t = YtY2, X2 = Y2 - YIY2' and

J=I -Yz Yz

) -Yl

Yt

Y2 .

TRANSFORMAT]ONS

209

Hence
fy t Y2(Yl, Y2)

= Y2

1 1 ,tilt +1I 2(YIY2t 1 - 1(Y2 - YIY2t 2- 1 e- AYl!ID(Yl, Y2) r(nl) r(n 2)

x [

r(nl

1I ;"lIt+ 2

+ n2 )

ylIl+1I2-le-AYl] (y)]. 2 (0,00) 2

It turns out that Yl and Y2 are independent and Y1 has a beta distribu-

tion with parameters n1 and n2

////

Of the three conditions that are imposed on the transformation Yl = gl(Xb X2) and Y2 = g2(Xl, X2), the sometimes restrictive condition that the transformation be one-to-one Can be relaxed. For the transformation Yl = gl(Xl, X2) and Y2 = g2(X 1 , x 2), each point in X will correspond to just one point in ~ ; but to a point in ~ there may correspond more than one point in X, which says that the transformation is not one-to-one and consequently Theorem 13 as stated is not applicable. If, however, X can be decomposed into a finite number of disjoint sets, say Xl' ... , Xm, so that Yl = gl(X 1 , x 2) and Y2 = g2(Xl, X2) define a one-to-one transformation of each Xi onto ~ then the joint density of Y1 = gl(X 1 , X 2) and Y2 = g2(X 1 , X 2) can be found. Let Xl = g;/(Yb Y2) and X2 = g;}(Yl' Y2) denote the inverse transformation of ~ onto Xi for j = I, ... , m, and set
a -1 9ti aYl agu -1 aYl
9ti -

-1

Ji =

-- -aY2

aY2 ag2i -1

Theorem 14 Let Xl and X 2 be two jointly continuous random variables with density function!xl,X2(Xl, X2)' Assume that X Can be decomposed into sets Xl' "', Xm such that the transformation Yl = gl (Xl' X2) and Y2 = g2(Xl, x 2) is one-to-one from Xi onto~. Let Xl = gl/(Yl' Y2) and X2 = g2"/(Yl' Y2) denote the inverse transformation of ~ onto Xi' i = 1, ... , m. Assume that all first partial derivatives of g~/ and g:;/ are continuous on ID and that J i does not vanish on~, i = I, ... , m. Then

210

DISTRIBUTIONS OF FUNCITONS OF RANDOM VARJABLES

fYI. yiYl' YZ)

i=l

L IJilfX.,X2(gl/(Yl' Yl), U2/(Yl' YZI\~,(YI' Yl)'

(41)

We illustrate this theorem with Example 26.

1/11

EXAMPLE 26 Assume that X I and X z are independent standard normal and random variables. Consider the transformation Yl = xi + Y 2 = Xl' which implies Xl = + J Yl - y~ and Xz = Yl so that the transformation is not one-to-one. Here X = {(Xl' Xz): - 00 < Xl < 00, - 00 < Xl < oo}, and ID ={(YH Yl): 0 < Yl < 00, -,}Y. < Y2 < J YI}' If X is decomposed into Xl and Xz, where Xl = {(Xl' X2): 0 Xl < 00, -00 < X2 < oo} and Xl = {(Xl' Xl): -00 < Xl < 0, -00 < X2 < oo} (in the terminology of Theorem 14, m = 2), then our transformation is oneto-one for Xi onto ID, i = 1,2. gll(YI' Y2) = J Yl - yi, andg 2 l(Yl' Yl) = 1 Y2; gI2 (y., Yl) = J Yl - yi, and g2i(Yh Y2) = Yz; so

xi

J1 =

f(Yl -

Yl)-~

.1.

iJull iJY2 = t(Yl I

yn- t ,

and
1 ( Yl = - 2)-t - Yl

Hence, fyhyiYh Y2) = [IJllfxl. x2 (uIl(Yh Y2), UZ/(Yh Yl + IJzlfxt.x/uli(Yl' Y2)' u2i(Yl' Y2]IID(Yh Y2)

for YI

JYI - y~

. -. e
2n

_tyl

0 and -JYl < Y2 < JYl'

Now

= ~ e- t ),lJarc sin Yl
2n
= ;"

,,/1.)
-../)'1

JYl

e-ty.(~ +~) = ~ e- b

for Y, > 0,

an exponential distribution.

/111

TRANSFORMATIONS

211

Theorems 13 and 14 can be generalized from n = 2 to n > 2. We wilI state the generalization of Theorem 14. (Theorem 13 is a special case of Theorem 14.)

Theorem 15 Let Xl' X 2 , , X rJ be jointly continuous random variables Let X = {(xv .. , xrJ): with density function IXh .... Xn(Xt' , xrJ)' Ix ..... x(x t "" xrJ) > O}. Assume that X can be decomposed into sets
Xl' .. , Xm such that Yl gl(Xt, .'" x,,), Y2 =giXl' , .. , x,,), ... , Y" = . grJ(Xl, ... , xrJ) is a one-ta-one transformation of Xi onto ID, i = I, ... , m. Let Xl = g1/(Yb ... , y,,), ... , XrJ = g;' 1 (Yl' ... Y,,) denote the inverse transformaTIOn of ID onto Xi' i = I, "', m. Define

9ti OYI o9u -1

Ji

og1/ -OY2 ogu t OY2

...

...

o -1 9ti oY"
1

0921 - - - - ... . .. oYt oY"

to '" ................. to

"

. . . ..

U9", oYrJ

-1

for i = 1, .. " m. Assume that all the partial derivatives in J i are continuous over ID and the determinant J i is nonzero, i = 1, ... , m. Then

i=1

L IJdfxl ..... x (9i/(Yt, ... , Y,,), ... , 9rJi \Yl' ... , Yn

n

(42)

for (Yl, ... , Y,,) in

ID

IIII

EXAMPLE 27 Let Xl, X 2 , and X 3 be independent standard normal random variables, Yl = Xl' Y2 = (Xl + x2)/2, and Y3 = (Xl + X2 + x3)/3. Then Xl = YI, X2 = 2Y2 - Yl' and x3 = 3Y3 - 2Y2; so the transformation is one-to-one. (m = I in Theorem 15.)
1
J=

-1

2 -2

o o
3

6.

. 212

DISfRIBUTIONS OF FUNCTIONS OF RANPOM VARIABLES

fYI. Y2, Y3(Yl' Y2' Y3)

= =

IJlfXt.x2,X3(X 1 , X2, X3)

6(/~;;) 3 exp{ -l[yi + (2y, 3

y,)'

+ (3Y3 -

2y,)'])

6 ()2J exp[ -!(2yi - 4y,y, + 8y~ -

12Y'Y3

+ 9y~)].

The marginal distributions can be obtained from the joint distribution; for instance,

f f fYI, Y2. Y3(Ytt Y2' Y3) dYl dY2 6()2J rooexp[ -!(6y~ - 12y, Y3 + 9y~)] [-!(2yi - 4y,y, + 2yi y ,) dy, x (rooeXP = .)2 ()2,,)' r", exp [-!( 6y~ - 12y, Y3 + 6 y~)]exp[ -!(3 y~)] dy,
00 00

-00

-00

)]d

= (J3/J2rr) exp[ - !y~];

that is, Y3 is normally distributed with mean 0 and variance

IIII

PROBLEMS
1

Let X h X 2 , and X3 be uncorrelated random variables with common variance a 2 Find the correlation coefficient between Xl + X 2 and X 2 + X 3 . (b) Let Xl and X 2 be uncorrelated random variables. Find the correlation coefficient between Xl + X 2 and X 2 - Xl in terms of var [Xl] and var [X2]. (c) Let Xl, X 2 , and X3 be independently distributed random variables with common mean p. and common variance a 2 Find the correlation coefficient between X 2 - Xl and X3 - Xl' 2 Prove Theorem 2. 3 Let X have c.d.f. F x (') = F(). What in terms of F(') is the distribution of XI[o. CXJ)(X) = max [0, Xl? 4 Consider drawing bal1s, one at a time, without replacement, from an urn containing M bal1s, K of which are defective. Let the random variable X( Y) denote the number of the draw On which the first defective (nondefective) ba11 is obtained. Let Z denote the number of the draw On which the rth defective baH is obtained. (a) Find the distribution of X.
(a)

PROBLEMS

213

(b)

Find the distribution of Z. (Such distribution is often cal1ed the negative hypergeometric distribution.) (c) Set M = 5 and K = 2. Find the jOint distribution of X and Y. Let Xl, , Xn be independent and identically distributed with common density
/x(x) = x- 2 I u , CXl)(X).

8
9

10

11

12

Set Y = min [Xl, . , X n ]. Does 8[X.] exist? If so, find it. Does 8[ Y] exist? If so, find it. Let X and Y be two random variables having finite means. (a) Prove or disprove: 8[max [X, Yl] > max [8[X], 8[ Y]]. (b) Prove or disprove: 8[max [X, Yl + min [X, Yl] = 8[X] + 8[ Y]. The area of a rectangle is obtained by first measuring the length and width and then multiplying the two measurements together. Let X denote the measured length, Y the measured width. Assume that the measurements X and Y are random variables with jOint probability density function given by Ix. l'(x, y) = klr.9L.1.ILJ(x)hsw.I.2WJ(Y), where Land Ware parameters satisfying L > W> 0 and k is a constant which may depend on Land W. (a) Find 8[XYl and var [XYl. (b) Find the distribution of XY. If X and Yare independent random variables with (negative) exponential distributions having respective parameters Al and A2 , find 8[max [X, Yl]. Projectiles are fired at the origin of an xy coordinate system. Assume that the point which is hit, say (X, y), consists of a pair of independent standard normal random variables. For two projectiles fired independently of one another, let (XI, Yl ) and (X2 , Y 2 ) represent the points which are hit, and let Z be the distance between them. Find the distribution of Z2. HINT: What is the distribution of (X2 - X I )2? Of (Y2 - y 1 )2? Is (X2 - X I )2 independent of (Y2 - y.)2? A certain explOsive device wiJ] detonate if anyone of n short-lived fuses lasts longer than .8 seconds. Let Xl represent the life of the ith fuse": It can be assumed that each Xi is uniformly distributed over the interval 0 to I second. Furthermor~ it can be assumed that the XI'S are independent. (a) How many fuses are needed (i.e., how large should n be) if one wants to be 95 percent certain that the device wiH detonate? (b) If the device has nine fuses, what is the average life of the fuse that lasts the longest? Suppose that random variable Xn has a c.d.f. given by [(n - l)/n] <l>(x) + (l/n)Fn(x), where <l> (.) is the c.d.f. of a standard normal and for each n Fn() is a c.d.f. What is the limiting distribution of Xn? Let X and Y be independent random variables each having a geometric distribution. *(a) Find the distribution of X/(X + Y). [Define X/(X + Y) to be zero if X+ y=o.] (b) Find the joint moment generating function of X and X + Y.

214

DISTRIBUTIONS OF FUNCTIONs OF RANPOM vARIABLES

13 Let Xl and X 2 be independent standard norma1 random variables. + X 2 and Y 2 = Xl + Xl.

(a)

Let Y l = Xl

Show that the joint moment generating function of Y l and Y 2 is exp [tfl(1 - 2t2)] for ex)

< t1 <

ex)

and -

ex)

<

t2

< !.

Find the correlation coefficient of Y l and Y 2 14 Let X and Y be independent standard normal random variables. Find the m.g.f. of XY. 15 Suppose that Xl and X 2 are independent random variable~ each having a standard normal distribution.
(b)
(a)

(b)

Find the joint distribution of (Xl + X 2)/,v'2 and (X2 ~ Xl)/vi. Argue that 2XI X 2 and Xl- Xl have the same distribution. HINT:
X 2 ~ Xl - 2
2 2 _

Xl

+ X 2 X 2 - Xl Vi Vi

16 A dry-bean supplier fi]ls bean bags with a machine that does not work very wen, and he advertises that each bag contains 1 pound of beans. In fact, the weight of the beans that the machine puts into a bag is a random variable with mean 16 ounces and standard deviation 1 Ounce. If a box contains 16 bags of beans: (a) Find the mean and variance of the weight of the beans in a box. (b) Find approximately the probabiHty that the weight of the beans in a box exceeds 250 ounces. (c) Find the probability that two or fewer underweight (less than 16 ounce) bags are in the box if the weight of beans in a bag is assumed to be normally distributed. 17 Numbers are selected at random from the interval (0, 1). (a) If 10 numbers are selected, what is the probability that exactly 5 are less than

!?
If 10 numbers are selected, on the average how many are less than 1? (c) If 100 numbers are selected, what is the probability that the average of the numbers is less than 1? 18 Let Xl denote the number of meteors that collide with a test satellite during the
(b)

ith orbit. Let Sn =

L Xl; that is, Sn is the total number of meteors that collides

1=1

with the satellite during n orbits. Assume that the XI'S are independent and identically dist~ibuted Poisson random variables having mean A. (a) Find I[Sn] and var [Sn]. (b) If n = 100 and A = 4, find approximately pISIOO > 440]. 19 How many light bulbs should you buy if you want to be 95 percent certain that you will have 1000 hours of light if each of the bulbs is known to have a lifetime that is (negative) exponentially distributed with an average life of 100 hours? (a) Assume that an the bulbs are burning simultaneously. (b) Assume that one bulb is used until it burns out andlhen it is replaced. etc.

PROBLEMS

215

20

(a)

If Xl, ... , XII are independent and identical1y distributed gamma random

variables, what is the distribution of Xl + ... XII? (b) If X., . " XII are independent gamma random variables and if X, has parameters ri and ", i 1, ... , n~ what is the distribution of Xl + ... + XII 1 21 (a) If Xl, . , XII are independent identically distributed geometric random variables, what is the distribution of Xl + . . . Xn 1 (b) If Xl" . , XII are independent identically distributed geometric random variables with density 0(1- (J)Jf- 1 l(l.z ....(x), what is the distribution of
Xl (c)

+ ... + XII?

If Xl, .. , XII are independent identically distributed negative binomial

*22

23 *24

25 26 27
28

random variables, what is the distribution of Xl + XII ? (d) If Xl, . , Xn are independent negative binomial random variables and if Xi has parameters ri and p, what is the distribution of Xl + ... + XII? Kitty Oil Co. has decided to drill for oil in 10 different locations; the cost of drilling at each location is $10,000. (Total cost is then $100,000.) The probability of finding oil in a given location is only "1, but if oil is found at a given location, then the amount of money the company will get selling oil (excludi ng the initial $10,000 deming cost) from that location is an exponential random variable with mean $50,000. Let Y be the random variable that denotes the number of locations where oil is found, and let Z denote the total amount of money received from selling oj] from an the locations. (a) Find 8[Z]. (b) Find P[Z> 100,0001 Y = 1] and P[Z > 100,0001 Y = 2]. (c) How would you find p[Z > 100,000]1 Is P[Z> 100,000] > i? If Xl, .. , X" are independent Poisson distributed random variables, show that the conditional distribution of Xl, given Xl + ... + X", is binomial. Assume that Xl, ... , Xk+l are independent Poisson distributed random variables ",,+1' Show that the conditional distribution with respective parameters of XI, ... , X" given that Xl ... + X"+1 = n has a multinomial distribution with parameters n, where" = + ... +",,+1' . If X has a uniform distribution over the interval ( -7T/2, 7T/2), find the distribution of Y tan X. If X has a normal distribution with mean f.L and variance (12, find the distribution, mean, and variance of Y = eX. Suppose X has c.d.f. Fx(x) exp [-e-(Jf-<l:)/P]. What is the distribution of Y exp [-(X - (X)/{1]? Let X have density 1 x l

"1, ... ,

"d", "" ""I",

"1

fx(x; a, b)

B(a, b) (l

+ Xf+b 1(0, ~)(x),

where a > 0 and b > O. (This density is often called a beta distribution of the second kind.) Find the distribution of Y = 1/(1 + X). 29 If X has a uniform distribution on the interval (0, 1), find the distribution of 11 X. Does 8[11 XJ exist? If so, find it.

216

DISTRIBUTIONS OF FUNCTIONS OF RANDOM VARIABLES

30

31 32 33

34 35 36
37

38

39
40 41

*42
43 *44

45 46 47 48 49

Give an example of a distribution of a random variable X for which 8[1/ Xl is not finite. (b) Give an example of a distribution of a random variable X for whiCh 8[1/ Xl is finite, and evaluate 8[1/ Xl. If Ix(x) = 2xe- X2 / to . oo)(x), find the density of Y = X2. If X has a beta distribution, what is the distribution of 1 - X? If Ix(x) = e-x/(o. oo)(x), find the distribution of X/(1 + X). If Ix(x) = 1/7T(1 + x 2), find the distribution of I/X. If Ix (x) = 0 for x <0, find the density of Y = aX2 + b in terms of Ix( .) for a > o. If X has the Weibu]] distribution as given in Eq. (42) of Chap. III, what is the distribution of Y = aXb ? (a) Let Y = X2 and Ix(x) = (1/8)/(0. 8)(X), 8> o. Find the c.dJ. of X and Y. Find the density of Y. (b) Let Y = X2 and Ix(x) = (!8)/(_8. 8)(X), 8> O. Find the c.dJ. and density of Y. If X and Yare independent random variables, each having the same geometric distribution, find the distribution of Y - X. If X and Yare independent random variables, each having the same negative exponential distribution, find the distribution of Y - X. If X, Y, and Z are independent random variables, each uniformly distributed over (0, 1), what is the distribution of XY/Z? Assume that X and Y are independent random variables, where X has a p.dJ. given by Ix(x) = 2xl(o, l)(x) and Y has a p.dJ. given by Iy(y) = 2(1 - y)/(o, 1)(Y). Find the distribution of X + Y. Let X and Y be independent Poisson distributed random variables. Find the distribution of Y - X. If Ix(x) = 1(0. l)(x), find the density of Y = 3X + 1. Let X and Y be two independent beta-distributed random variables. Is XYalways beta-distributed? If not, find conditions on the parameters of X and Y that wi1l imply that XY is beta-distributed. If Ix. y(x, y) = e-(XH)/(o. oo)(x)/(o. OO)(y), find the density of Z = (X + Y)/2.
(a)

If lx, y(x, y) =4xye-(.%2 U 2)/(0. oo)(x)/(o. oo)(Y), find the density of V X2 + 1'2. If Ix. y(x, y) = 4xyl(o, 1)(x)/(o, l)(y)' find the joint density of X2 and y2. If Ix. y(x, y) = 3x/(0.x)(y)/(o, 1)(X), find the density of Z = X - Y. If Ix(x) = [(1 + x)/2l/(-I,1)(X), find the density of Y = X2. 50 If Ix. y(x, y) = 1(0. 1)(x)/(o, l)(Y), find the density of Z, where
51 Y - 1)/(1. OO)(X + y). If lx, y(x, y) = e-(x+)')/(o, oo)(x)/(o. oo)(y), find the joint density of X and X + Y. If Ix. y. z(x, y, z) = e-(:K+),H)/(o, oo)(x)/(o. oo)(y)/(o, oo)(z), find the density of their Z

= (X + Y)/(-oo. 1](X+ y) + (X +

52

average (X + Y + Z)/3. 53 If Xl and X 2 are independent and each has probability density given by Ae-Ax/(o. oo)(x), find the joint distribution of YI = X I /X2 and Y 2 = Xl + X 2 and the marginal distributions of YI and Y 2

PROBLEMS

217

*54 Let Xl and X 2 be independent random variables, each normally distributed with parameters p. = 0 and (72 = 1. Find the joint distribution of Y1 = Xl + Xl and Y 2 = X 1 !X2 Find the marginal distribution of Y1 and of Y 2 Are Y1 and Y 2 ( independent? 55 If the joint distribution of X and Y is given by

Ix. rex, y) = 2e- (.x'+.v) I[o.),](x)I[o. C;C)(Y},

find the jOint distribution of X and X + Y. Find the marginal distributions of X and X Y. Let Ix. rex, y) K(x + y)l(o. l)(x)/(o. l)(y)I(o. l)(x y). (a) Find Ix( '). (b) Find the jOint and marginal distributions of X + Yand Y - X. Suppose !c.x. ]I) I z(x, y Iz) = [z + (I - z}(x+- y)I(o. l)(x)l(o. l)(Y) for 0 <z < 2, and Iz(z} = !i[o. 2](Z}. (a) Find 8[X YJ. (b) Are X and Y independent? Verify. (e) Are Xand Z independent? Verify. (d) Find the joint distribution of X and X + Y. (e) Find the distribution of max [X, Y]IZ = z. (I) Find the distribution of (X + Y) I Z = z. A system will function as long as at least one of three components functions. When all three components are functioning, the distribution of the life of each is exponential with parameter iA, When only two are functioning, the distribution of the I ife of each of the two is exponential with parameter !A; and when only One is functioning, the distribution of its 1ife is exponential with parameter A. (a) What is the distribution of the lifetime of the system? (b) Suppose nOw that only one component (of the three components) is used at a time and it is replaced when it fails. What is the distribution of the lifetime of such a system? The system in the sketch will function as long as component C 1 and at least One of the components C 2 and C 3 functions. Let Xi be the random variable denoting the lifetime of component C" i I, 2, and 3. Let Y = max [X2 X 3 ] and Z = min [Xl, Assume that the Xs's are independent (negative) exponential random variables with mean I. (a) Find 8[Z] and var [Z]. (b) Find the distribution of the lifetime of the system.

56

57

58

59

n.

r--

C2

:-

CI

c3

I-

218

DISTRIBUTIONS OF FUNCfIONS OF RANDOM VARIABLES

A system, which is composed of two components~ will function as 19n9 as at least one of the two components functions. When both components are operating~ the lifetime distribution of each is exponential with mean 1. However~ the distribution of the remaining lifetime of the good component~ after ~ne fails, is exponential with mean t. (The idea is that after one component fails the other component carries twice the load and hence has only half the expected lifetime.) Find the lifetime distribution of the system. *61 Suppose that (X, Y) has a bivariate normal distribution. Find the jOint distribution of aX + bY and eX + dY for constants a, b, e, and d satisfying ad-be#O. Find the distribution of aX + bY. HINT: Use the moment-generating-function technique and see Example 7. 62 Let Xl and X 2 be independent standard normal random variables. Let U be independent of Xl and X 2 , and assume that U is uniformly distributed over(O, I}. DefineZ = UXI + (1 - U}X2 (a) Find the conditional distribution of Z given U = u. (b) Find cB'[Z] and var [Z]. * (e) Find the distribution of Z.
60

VI
SAMPLING AND SAMPLING DISTRIBUTIONS

1 INTRODUCTION AND SUMMARY

The purpose of this chapter is to introduce the concept of sampling and to present some distribution theoretical results that are engendered by sampling. It is a connecting chapter-it merges the distribution theory of the first five chapters into the statistical theory of the last five chapters. The intent is to present here in one location some of the laborious derivations of distributions that are associated with sampling and that will be necessary in our future study of the theory of statistics, especially estimation and testing hypotheses. Our thinking is that by deriving these results now, our later presentation of the statistical theory will not have to be interrupted by their derivations. The nature of the material to be given here is such that it is not easily motivated. Section 2 begins with a discussion of populations and samples. It ends with the definitions of statistic and of sample moments. Sample moments are important and useful statistics. Section 3 is devoted to the consideration of various results associated with the sample mean. The law of large numbers and the central-limit theorem are given, and then the exact distribution of the sample means from several of the different parametric families of distributions

220

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

introduced in Chap. III is given. Sampling from the normal distribution is considered in Sec. 4!J where the chi-square, F, and t distributions are defined. Order statistics are discussed in the final section; theY!J like sample moments, are important and useful statistics.

2 SAMPLING
2 .. 1 Inductive Inference
Up to now we have been concerned with certain aspects of the theory of probability, including distribution theory. Now the subject of sampling brings us to the theory of statistics proper, and here we shall consider briefly one important area of the theory of statistics and its relation to sampling. Progress in science is often ascribed to experimentation. The research worker performs an experiment and obtains some data. On the basis of the . data, certain conclusions are drawn. The conclusions usually go beyond the materials and operations of the particular experiment. In other words, the scientist may generalize from a particular experiment to the class of all similar experiments. This sort of extension from the particular to the general is called inductive inference. It is one way in which new knowledge is found. Inductive inference is well known to be a hazardous process. In fact, it is a theorem of logic that in inductive inference uncertainty is present. One simply cannot make absolutely certain generalizations. However, uncertain inferences can be made, and the degree of uncertainty can be measured if the experiment has been performed in accordance with certain principles. One function of statistics is the provision of techniques for making inductive inferences and for measuring the degree of uncertainty of such inferences'. Uncertainty is measured in terms of probability, and that is the reason we have devoted so much time to the theory of probability. Before proceeding further we shall say a few words about another kind of inference-deductive inference. While conclusions which are reached by inductive inference are only prObable, those reached by deductive inference are conclusive. To illustrate deductive inference, consider the following two statements: One of the interior angles of each right triangle equals 90, (ii) Triangle A is a right triangle.
(i)

If we accept these two statements, then we are forced to the conclusion:

(iii)

One of the angles of triangle A eq uals 90.

SAMPLING

221

FIGURE 1

West Point graduates

This is an example of deductive inference, which can be described as a method of deriving information [statement (iii)] from accepted facts [statements (i) and (ii)]. Statement (i) is called the major premise, statement (ii) the minor premise, and statement (iii) the conclusion. For another example, consider the following: (i) (ii) (iii) Major premise: All West Point graduates are over 18 years of age. Minor premise: John is a West Point graduate. Conclusion: John is over 18 years of age.

West Point graduates is a subset of all persons over 18 years old, and John is an element in the subset of West Point graduates; hence John is also an element in the set of persons who are over 18 years old. While deductive inference is extremely important, much of the new knowledge in the real world comes about by the process of inductive inference. In the science of mathematics, for example, deductive inference is used to prove the'orems, while in the empirical sciences inductive inference is used to find new knowledge. Let us illustrate inductive inference by a simple example. Suppose that we have a storage bin which contains (let us say) 10 million flower seeds which we know will each produce either white or red flowers. The information which we want is: How many (or what percent) of these 10 million seeds will produce white flowers? Now the only way in which we can be sure that this question is answered correctly is to plant every seed and observe the number producing white flowers. However, this is not feasible since we want to sell the seeds. Even if we did not want to sell the seeds, we would prefer to obtain an answer without expending so much effort. Of course, without planting each seed and observing the color of flower that each produces we cannot be certain of the number of seeds producing white flowers. Another thought which occurs is: Can we plant a few of the seeds and, on the basis of the colors of these few flowers, make a statement as to' how many of the 10 mi11ion seeds will produce

222

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

white flowers? The answer is that we cannot make an exact prediction as to how many white flowers the seeds will produce but we can make a probabilistic statement if we select the few seeds in a certain fashion. This is inductive inference. We select a few of the 10 million seeds, plant them, observe the number whiCh produce white flowers, and on the basis of these few we make a prediction as to how many of the 10 million will produce white flowers; from a knowledge of the color of a few we generalize to the whole 10 million. We cannot be certain of our answer, but we can have confidence in it in a frequencyratio-probability sense.

2.2 Populations and Samples

We have seen in the previous subsection that a central problem in discovering new knowledge in the real world consists of observing a few of the elements under discussion, and on the basis of these few we make a statement about the totality of elements. We shall now investigate this procedure in more detail.

Definition 1 Target population The totality of elements which are under discussion and about which information is desired will be called the target
population. // / /

In the example in the previous subsection the 10 million seeds in the storage bin form the target population. The target population may be all the dairy cattle in Wisconsin on a certain date, the prices of bread in New York City on a certain date, the hypothetical sequence of heads and tails obtained by tossing a certain coin an infinite number of times, the hypothetical set of an infinite number of measurements of the velocity of light, and so forth. The important thing is that the target population must be capable of being quite well defined; it may be real or hypothetical. . The problem of inductive inference is regarded as follows from the point of view of statistics: The object of an investigation is to find out something about a certain target population. It is generally impossible or impractical to examine the entire population, but one may examine a part of it (a sample from it) and, on the basis of this limited investigation, make inferences regarding the entire target population. The problem immediately arises as to how the sample of the population should be selected. We stated in the previous section that we could make probabilistic statements about the population if the sample is selected in a certain fashion. Of particular importance is the case of a simple random sample, usually called a random sample, which is defined in Definition 2 below for any

SAMPLING

223

population which has a density. That is, we assume that each element in our population has some numerical value associated with it and the distribution of these numerical values is given by a density. For such a population we define a random sample.

Definition 2 Random sample have a joint density fXJ' .__ Xn( "

Let the random variables Xl' X 2, , .) that factors as follows:

.. ,

Xn

fXJ.Xl .... Xn(Xh X2, "', xn)

= f(Xl)f(X2)

..... f(xn),

where f( .) is the (common) density of each Xi' Then Xl' X 2, Xn is defined to be a random sample of size n from a population with density f(). ///1 In the example in the previous subsection the 10 million seeds in the storage bin formed the population from which we propose to sample. Each seed is an element of the population and will produce a white or red flower; so, strictly speaking, there is not a numerical value associated with each element of the population. However, if we, say, associate the number 1 with white and the num ber 0 with red, then there is a numerical value associated with each element of the population, and we can discuss whether or not a particular sample is random. The random variable Xi is then 1 or 0 depending on whether the ith seed sampled produces a white or red flower, i 1, ... , n. Now if the sampling of seeds is performed in such a way that the random variables Xl' ... , Xn are independent and have the same density. then, according to Definition 2, the sample is caned random. An important part of the definition of a random sample is the meaning of the random variables Xl, .. " X n The random variable Xi is a representation for the numerical value that the ith item (or element) sampled will assume. After the sample is observed, the actual values of Xl' ... , Xn are known, and as usual, we denote these observed values by X h . , X n Sometimes the observations Xl' . , Xn are called a random sample if Xl, ... , Xn are the values of Xl, .. " X n where Xl' "', Xn is a random sample. Often it is not possible to select a random sample from the target population, but a random sample can be selected from some related population. To distinguish the two populations, we define sampled population.

Definition 3 Sampled population Let Xl' X 2, ... , Xn be a random samp1e from a population with density f( . ); then this population is called the sampled population.' IIII

224

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

Valid probability statements can be made about sampled populations on the basis of random samples, but statements about the target populations are not valid in a relative-frequency-probability sense unless the target population is also the sampled population. We shaH give some examples to bring out the distinction between the sampled population and the target population. EXAMPLE I Suppose that a sociologist desires to study the religious habits of 20-year-old males in the United States. He draws a sample from the 20-year-old males of a large city to make his study. In this case the target population is the 20-year-old males in the United States, and the sampled population is the 20-year-old males in the city which he sampled. He can draw valid relative-frequency-probabilistic conc1usions about his sampled population, but he must use his personal judgment to extrapolate to the target population, and the reliability of the extrapolation cannot be measured in relative-frequency-probability terms. 111/ EXAMPLE 2 A wheat researcher is studying the yield of a certain variety of wheat in the state of Colorado. He has at his disposal five farms scattered throughout the state on which he can plant the wheat and observe the yield. The sampled population consists of the yields on these five farms, whereas the target population consists of the yields of wheat on IIII every farm in the state. This book will be concerned with the problem of selecting (drawing) a sample from a sampled population with density f( .), and on the basis of these sample observations probability statements will be made about f( .), or inferences about f( . ) will be made.
Remark We shall sometimes use the statement "population f()" to mean" a population with density f( . )." When we use the word" population" without an adjective" sampled" or "target," we shall always mean IIII sampled population.

2.3

Distribution of Sample Definition 4 Distribution of sample Let Xl' X 2 , , Xn denote a sample of size n. The distribution of the sample Xl' ... , Xn is defined to be the joint distribution of X h ... , X n II/I

Suppose that a random variable X has a density 1(') in some population, and suppose a sample of two values of X, say Xl and X2' is drawn at random.

SAMPLING

225

is called the first observation, and X2 the second observation. The pair of numbers (Xl' x 2 ) determines a point in a plane, and the collection of all such pairs of numbers that might have been drawn forms a bivariate population. We are interested in the distribution (bivariate) of this bivariate population in terms of the original density f( . ). The pair of numbers (Xl' X2) is a value of the joint random variable (Xl' X 2), and Xl' X 2 is a random sample (of size 2) from I( ,). By definition of random sample, the joint distribution of Xl and X 2 , which we call the distribution of our random sample of size 2, is given by fxt, X2(X l , X2) = f(x l )f(x 2) As a simple example, suppose that X can have only two values, 0 and 1, with probabilities q = 1 - p and p, respectively. That is, X is a discrete random variable which has the Bernoulli distribution
Xl

f(x) = pXql -x/{O.l}(X),

(1)

The joint density for a random sample of two values fromf( .) is IXI. X2(X l , X2) = f(x l )f(x 2) = pXI +X2q2- XI-X2f{O. l}(xl)I{o. 1}(x 2).

(2)

It is to be observed that this (bivariate) density is not what we obtain as the distribution of the number of successes, say Y, in drawing two elements from a Bernoulli population. The density of Y is given by
frey)

G)

p'q 2-,

for y = 0, 1,2.

The single random variable Yequals Xl + X 2 . It should be noted thatfxl. X2(X t , X2) gives us the distribution of the sample in the order drawn. For instance, in Eq. (2), f X Io X 2(O, 1) = pq refers to the probability of drawing first a 0 and then a 1. Our comments for a random sample of size 2 generalize to a random sample of size n, and wehave the following remark.

Remark If Xl' X 2 , ... , Xn is a random sample of size n. from f( ,), then the distribution of the random sample Xl' .. " X n , defined as the joint distribution of X h . . . , X n, is given by fxt. .... x,.(x l , ... , xn) = f(x l )'" 'f(xn)'

1//1
Note that again this gives the distribution of the sample in the order drawn. Also, note that if X h ... , Xn is a random sample, then Xl' ... , Xn are stochastically independent. We might further note that ou~ definition of random sampling has automatical1y ru]ed out sampling from a finite population without replacement s"ince, then, the results of the drawings are not independent.

 226

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

2.4 Statistic and Sample Moments

One of the central problems in statistics is the following: It is desired to study a population which has a density I( . ; 0), where. the form of the density is known but it contains an unknown parameter 0 (if 0 is known. then the density func tion is completely specified). The procedure is to take a random sample Xl' X 2 , XII of size n from this density and let the value of some function, say t(XI' X2 , . , XII)' represent or estimate the unknown parameter O. The pr<?blem is to determine whiCh function will be the best one to estimate O. This problem will be formulated in more detail in the next chapter. In this section we shall examine certain functions, namely, the sample moments, of a random sample. First, however, we shall define what we shall m.ean by a statistic.

Definition 5 Statistic A statistic is a function of observable random variables, which is itself an observable random variable, which does not 1/11 contain any unknown parameters.
The qualification imposed by the word" observable" is required because of the way yve intend to use a statistic. (" Observable" means that we can observe the values of the random variables.) We intend to use a statistic to make inferences about the density of the random variables. ~nd if the random variables were not observable, they would be of no use in making inferences. For example, if the observable random variable X has the density Jl.<F2 (x), where J1 and a 2 are unknown, then X - J1 is not a statistic; neither is X/a (since they are not functions of the observable random variable X only-they contain unknown parameters), but X, X + 3, and X2 + log X 2 are statistics. In the formulation above, one of the central problems in statistics is to find a suitable statistic (function of the random variables Xl' X 2 , , Xn) to represent O.

EXAMPLE 3 If X B .. , Xn is a random sample from a density I( . ; 0), then (provided Xl' ... , XII are observable)

is a statistic, and !{min [Xh ... , XII]

..

xn = - I x,
n
i= I

+ max [Xl' ... , XII]}

0 is not a

is also a statistic. If I(x; 0) o, 1 (x) and 0 is unknown, Xn statistic since it depends on 0, which is unknown.

////

SAMPLING

227

Next we shall define and discuss some important statistics, the sample moments. Sample moments Let Xl' X 2 , , Xn be a randonl sample from the density f( . ). Then the rth sample moment about 0, denoted ~y M;, is defined to be

Definition 6

M;=n

i= I

LXi

(3)

In particular, if r = 1, we gf!t the sample mean, which is usually denoted by X or Xn; that is,

Also, the rth sample moment about X n , denoted by M,., is defined to be

_)' M,. = -1 ~ ~ (Xi - Xn .

n
i= 1

(5)

IIII
Remark Note that sample moments are examples of statistics.

IIII

We will consider in detail some properties of the sample mean in Sec. 3 below. In Chap. II we defined the rth moment of a random variable X, or the rth moment of its corresponding density fx(), to be @[X'] = /1;. We could say that 6"[XY] is the rth population moment of the population with density f(x) = fx(x). We_ shall now show that the sample moments reflect the population moments in the sense that the expected value of a sample moment (about 0) equa}s the corresponding population moment. Also, the variance of a sample moment wiJI be shown to be (lIn) ti~es some function of population moments. The implication is that for a given population the values that the sample moment assume will tend to be more concentrated about the corresponding population moment for large sample size n than for small sample size.' Thus a sample moment can be used to estimate its corresponding population moment (provided the population 'moment exists).

---

Xn be a random sample from a population with a density f() The expected value of the rth sample moment (about 0) is equal to the rth population moment; that is,
,

Theorem 1 Let Xl' X 2

(if J.l; exists).

(6)

228

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

Also, (if /1;, exists).

PROOF

(7)

var[M;] = var

[~

ni=1

t X~]
,In

(~)2 var [.tx~]

n

(!)2 .t var[X~]
,=1

In particular, if r

1, we get the following coro]]ary.

. .

1111

Corollary Let Xl' X 2

Xn be a random sample from a density I( . ),

I " and let Xn = - L Xi be the sample mean; then

n i== 1

and where It and

(J2

- ] = -1 (J 2 , var [ Xn
n

(8)

are, respective1y, the mean and variance of I( .).

1111

As we mentioned earlier, properties of the sample mean wi11 be studied in detail in the next section. Theorem 1 gives the mean and variance in terms of population moments of the rth sample moment about 0; a s!milar, though more complicated, result can be derived for the mean and variance of the rth sample moment about the sample mean. We wiJ] be content with looking only at the particular case
r

= 2, that is, at M2 = (lIn) L (Xi - X)2.

i= 1

M2 is sometimes cal1ed the sample

v!!~iaI}.9..e..;

although we will take Definition 7 as our definition of the sample

varIance.

SAMPLING

229

Definition 7 Sample variance Let Xl) X 2 , from a 9-ensity f( . ); then

Xn be a random sample

for is defined to be the sample variance.

n> 1

(9)

/III

The reason for taking 8 2 rather than M 2 as our definition of the sample variance (both measure dispersion in the sample) is that the expected value of 8 2 equals the population variance. The proof of the following remark is left as an exercise.

Remark

nn I 2 = 8 = 2 ( 8n 1) '" L, '" L, (Xi - Xj) 2 . nni= 1 j= 1

2

1111

Theorem 2
f( .), and let

Let Xh X 2 ,

... ,

Xn be a random sample from a density

Then and var [8 2 ] for n > 1.

(10)

(Only the first part wil1 be proved.) Recall that (J2 = 8[(X - 11)2] and 11,. 8[(X 11)"]. We commence by noting and proving an identity that is quite useful.
PROOF

i= 1

L (Xi +

11)2 =

L1(Xi -

x)2

+ n(X -

11)2

(11)

sInce

L (Xi -

11)2

= L (Xi

- X

X - 11)2

L [(Xi -

X)

+ (X -

11)]2 Il)2

= L [(Xi - X)2 + 2(X, - X)(X -11) + (X - 11)2]

= L (Xi - X)2

= L (Xi

- X)2

+ 2( X + n(X -

11) L (Xi - X) tt)2.

+ n(X -

230

SAMPUNG AND SAMPLING DISTIUBUTIONS

VI

Using the identity of Eq. (11), we obtain

S[8 2 ] = S[
=

n-1

L (Xi -

X)2]

1 sf (Xi - p)2 - n(X - p)2] n - 1 L=l

n

~ 1 {.t1 8 [(X, - p)2] - n8[(X - p)2]}

-

n~ 1 ttu nvar[X]}
n -1
n (12) =

= 1 ( n(12 -

(12.

Although the derivation of the formula for the variance of 8 2 can be accomplished by utilizing the above identity [Eq. (11)] and

_ X- p

= - L. Xi - n
= -

1,.

np

111

L Xi - n - L p = - L (Xi n n

p),

such derivation is lengthy and is omitted here only to be relegated to the Problems. / / // Sample moments are examples of statistics that can be used to estimate their population counterparts; for example, M; estimates It;, X estimates It, and 8 2 estimates (12. In each case, we are taking some function of the sample, which we can observe, and using the value of that function of the sample to estimate the unknown population parameter.

SAMPLE MEAN

The first sample moment is the sample mean defined to be

= X" = -

I "

ni=l

L Xi'

where Xh X 2 , , X,. is a random sample from a density I( .). X is a function of the random variables Xl' ... , X,., and hence theoretically the distribution of X can be found. In general, one would suspect that the distribution of X

SAMPLE MEAN

231

depends on the density f() from which the random sample was selected, and indeed it does. Two characteristics of the distribution of X, its mean and variance, do not depend on the density f( . ) per se but depend on]y on two characteristics of the density f(). This idea is reviewed in the following subsection, while succeeding subsections consider other results involving the samp]e mean. The exact distribution of X will be given for certain specific densities f( . ). It might be helpful in reading this section to think of the sample mean X as an estimate of the mean J1 of the density f() from which the samp]e was selected. We might think that one purpose in taking the sample is to estimate J1 with X.

3.1

Mean and Variance Theorem 3 Let Xl' X 2 ,

,

Xn be a random sample from a density f(),

(12,

which has mean J1 and finite variance and

and let X = (lIn)

Xi.

Then
(12)

i=l

IIII
Theorem 3 is just a restatement of the corol1ary of Theorem 1. In light of using a va]ue of X to estimate J1, let us note what Theorem 3 says. @[ X] = J1 says that on the average X is equal to the parameter J1 being estimated or that the distribution of X is centered about J1. var [Xl = (1ln)(12 says that the spread of the va]ues of X about J1 is sma]] for a ]arge samp]e size as compared to a small sample size. For instance, the variance of the distribution of X for a sample of'size-W is one-ha]f the variance of the distribution of X for a sample of size 10. So for a large sample size the values of X (which are used to estimate J1) tend to be more concentrated about J1 than for a smal1 sample size. This notion is further exemplified by the law of large numbers considered in the next subsection.

3.2

Law of Large Numbers

Letf( ; (J) be the density of a random variable X. We have discussed the fact that one way to get some information about the density function f( ; (J) is to observe a random sample and make an inference from the sample to the popu]ation. If (J were known, the density functions would be completely specified,

"'"

--

232~AMPLING
-...-'

AND SAMPLING DISTRIBUTIONS

VI

and no inference from the sample to the popUlation would be necessary. Therefore, it seemS that we would like to have the random sample tell us something about the unknown parameter 0. This problem will be discussed in detail in the next chapter. In this subsection we shall discuss a related particular problem. Let S[X] be denoted by J-l in the density f(). The problem is to estimate J-l. In a loose sense, 8[X] is the average of an infinite number of values of the random variable X. In any real-world problem we can observe only a finite number of values of the random variable X. A very crucial question then is: Using only a finite number of values of X (a random sample of size n, say), can any reliable inferences be made about S[X], the average of an infinite number of values of X? The answer is "yes"; reliable inferences about S[X] can be made by using only a finite sample, and we shaH demonstrate this by proving what is called the weak law of large numbers. In words, the law states the following: A positive integer n can be determined such that if a random sample of size n or larger is taken from a population with the density 1(') (with 8[X] = 11), the probability can be made to be as close to I as desired that the sample mean X will deviate from J-l by less than any arbitrarily specified small quantity. More precisely, the weak law of large numbers states that for any two chosen small numbers e and c5, where e > 0 and 0 < c5 < 1, there exists an integer n such that if a random sample of size n or larger is obtained from f() and the sample mean, denoted by Xn , computed, then the probabiJity is greater than I - c5 (i.e., as close to I as desired) that Xn deviates from 11 by less than e (i.e., is arbitrarily close to J-l). In symbols this is written: For any e> 0 and 0 < b < I there exists an integer n such that for all integers m> n

The weak law of large numbers is proved using the Chebyshev inequality given in Chap. II.

Theorem 4 Weak law of large numbers Let I( . ) be a density with mean

J-l and finite variance (J2, and let xn be the sample mean of a random

sample of size n from f(')' Let e and c5 be any two specified numbers satisfying e > 0 and 0 < b < 1. If n is any integer greater than (J2je 2 b, then
P[ - e < Xn - J-l < e] > I - b.
PROOF

(13)

Theorem 5 in Subsec. 4.4 of Chap. II stated that P[g( X) k] < cB'[g(X)]/k for every k> 0, random variable X, and nonnegative function g(.). Equivalently, P[g(X) < k] > 1 - 8[g(X)]/k.

SAMPLE MEAN

233

Letg(X) = (Xn - J.l)2 and k =

2 8 ;

then

P[ -8 < Xn - J.l < 8] = P[I Xn - J.l1 < 8]

= P[ I Xn

- J.l1

<

8 ]

>-

S[(Xn - J.l?]
8
2

IIII
Below are two examples to illustrate how the weak law of large numbers can be used.

EXAMPLE 4 Suppose that some distribution with an unknown mean has variance equal to 1. How large a sample must be taken in order that the probability wi1l be at least .95 that the sample mean Xn will lie within .5 of the population mean? We have a 2 = 1, 8 = .5, and () = .05; therefore
a2 1 n > {)8 2 = .05(.5)2
=

80.

IIII

EXAMPLE 5 How large a sample must be taken in order that you are 99 percent certain that Xn is within .5a of J.l? We have 8 = .5a and () = .01. Thus

IIII
We have shown that by use of a random sample inductive inferences to populations can be made and the reliability of the inferences can be measured in terms of probability. For instance, in Example 4 above, the probability that the sample mean will be within one-half unit of the unknown population mean is at least .95 if a sample of size greater than 80 is taken.

3.3

Central-limit Theorem

Although we have already stated the central-limit theorem in our study of distribution theory in Chap. V, we will repeat it here in our study of the sample mean X because it gives the asymptotic distribution of X. At the outset of this

~~~PLING AND SAMPLING DISTRIBUTIONS

VI

section we indicated that we were interested in the distribution of x. The central-limit theorem, which is one of the most important theorems in all of probability and statistics, tells us approximately how X is distributed.

Theorem 5 Central-limit theorem Let/(') be a density with mean J.! and finite variance a 2 Let X" be the sample mean of a random sample of size n from/(). Let the random variable Z" be defined by
Z = X" - 8[X,,] = X" - J.!

"

Jvar [X,,]

alJn

(14)

Then, the distribution of Z" approaches the standard normal distribution IIII as n approaches infinity. Theorem 5 tells us that the limiting distribution of Z" (which is X" standardized) is a standard normal distribution, or it tells us that X" itself is approximately, or asymptotically, distributed as a norma] distribution with mean Jl and variance a2 In. The astonishing thing about Theorem 5 is the fact that nothing is said about the form of the original density function. Whatever the distribution function, provided only that it has a finite variance, the sample mean wiJ] have approximately the normal distribution for large samples. The condition that the variance be finite is not a critical restriction so far as applied statistics is concerned because in almost any practical situation the range of the random variable will be finite, in which case the variance must necessarily be finite. The importance of Theorem 5, as far as practical applications are concerned, is the fact that the mean X" of a random sample from any distribution with finite variance a 2 and mean J.! is approximately distributed as a normal random variable with mean J.! and variance a2 /n. We shall not be able to prove Theorem 5 because it requires rather advanced mathematical techniques. However, in order to make the theorem plausible, we shall outline a proof for the more restricted situation in which the distribution has a moment generating function. The argument will be essentially a matter of showing that the moment generating function for the sample mean approaches the moment generating function for the normal distribution. Recall that the moment generating function of a standard normal dis t2 t2 tribution is given by e-t (See Subsec. 3.2 of Chap. III.) Let m(t) = e1: Let mzJt) denote the moment generating function of Z" . It is our purpose to show that mz.,(t) must approach met) when n, the sample size, becomes large.

SAMPLE MEAN

235

Now
mz.(t) = .'le'z"] = 8[exp tZ.] =

c[exp (t ~/J:)] .r[exp (~L :~J:)] = c9'[Ii exp (! .Xi - 11)] = .11 [ex p (Jt _ .Xi a- 11)] n ujJn n
=

&

i=1

1=1

using the independence of Xl' ... , X n Now if we let Y j = (Xi - II}jO', then mylt), the moment generating function of Y j , is independent of i since a II Yt have the same distribution. Let my(t) denote my.(t); then

Hence,
mz.(t) = [my

C~)

(15)

The rth derivative of my(tj J;;) evaluated at t = 0 gives

uS

the rth moment about

the mean of the densityf() divided by (O'J~y, so we may write

my (

In

t )

=1+~

III t

In + 2!
2

1 112 ( t )
0'2

In

1 113 ( t ) 3 + 3! 0'3 J~ + "',

(16)

and since III

= 0 and 112 = 0'2,

this may be written

my (

t ) J= 1+ 1 - (1 -2 t n n

1 113 J 3 t 3! nO'

I 114 4 ) + 4,4 t +... . .n a

(17)

Now lim (1

+ ujnt = ett2 ,

where u represents the expression within the paren-

theses in Eq. (17).

We have lim m ZI1(t) = ett \ so that in the Jimit, Zn has the

same moment generating function as a standard normal and, by a theorem similar to Theorem 7 in Chap. II, has the same distribution. The degree of approximation depends, of course, on the sample size and on the particular density f(). The approach to normality is iI1ustrated in Fig. 2 for the particular function defined by f(x) = e- X l(o,oo)(x). The solid curves give the actual distributions, while the dashed curves give the normal approximations. Figure 2a gives the original distribution which corresponds to samples of 1 ; Fig. 2b shows the distribution of sample means for n = 3; Fig. 2c

236 ')AMPLING

AND SAMPLING DISTRIBUTIONS

VI

"---~
1.5
1.0
n

=1

1.5
1.0

n =

1.5

n = 10

1
(a)

2 x

2 x

-'--------'------"-- x

(b)
FIGURE 2

(c)

gives the distribution of sample means for n = 10. The curves rather exaggerate the approach to norma1ity because they cannot show what happens on the tails of the distribution. Ordinarily distributions of samp1e means approach normality fair1y rapid1y with the sample size in the region of the mean, but more s10wly at points distant from the mean; usual1y the greater the distance of a point from the mean, the more slowly the normal approximation approaches the actual distribution. In the fol1owing subsections we wi]] give the exact distribution of the sample mean for some specific densities f(')'

3.4

Bernoulli and Poisson Distributions

If Xl' X 2 , , Xn is a random sample from a Bernoulli distribution, we can find the exact distribution of X n. CVVe know that Xn is approximately normally distributed.) The density from which we are sampling is

We know (see Example 9 of Chap. V) that that is,

L Xi
1

has a binomial distribution;

hence, the distribution of Xn is given by

p [ X.

= :]

(~)p'r'

for k = 0, I, ... , n.

(18)

SAMPLE MEAN

237

So Xn , the sample mean of a random sample from a Bernoulli density, takes on the values 0, lin, 21n, ... , 1 with respective binomial probabilities

2 p q (n)

2 n-2

, ... ,

If Xl' ... , Xn is a random sample from a Poisson distribution with mean A, then L: Xi also has a Poisson distribution with parameter nA (see Example 10 of Chap. V), and hence

P X n =- =P L:Xj=k n ;=1

_ k]

=-----k!

e - nA( nA)k

for

k = 0, 1, 2, _.. ,

(19)

which gives the exact distribution of the sample mean for a sample from a Poisson density.

3.S

Exponential Distribution
_,

Let X h X 2 ,

Xn be a random sample from the exponential density I(x) =

ee- 8xI(o. co)(x).

According to Example II of Chap_ V, parameters nand

" . L Xi has
1

a gamma distribution with

e; that is,
f :EXt () Z 1 r(n)
Z

,,-lene - 8z1(0, co) (Z, )

or for y > 0, and so for y > 0. Or,

P[X" < xl =

f(j
x

nx

1 r(n) z" lW'e- 9z dz 1

fo r(n)

- - (nu)"-le ne- n8u n du'

that is, X" has a gamma distribution with parameters n and nO.

238

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

3.6 Uniform Distribution

Let Xh ... , Xn be a random sample from a uniform distribution on the interval (0, I]. The exact density of Xn is given by

Ix.(x)

:~: (n ~ I)! [(nxr-

(7)(nx - 1)" + G)(nx - 2r- ' - .. , + ( -It(:)(nx- k)n-I] [(kin, )/nj(X).

I I

(HI

(20)

The derivation of the above (using mathematical induction and the convolution formula) is rather tedious and is omitted. Instead let us look at the particular cases n = I, 2, 3.

lx/x) = IX{x) = l(o,l](x), Ix 2 (x)

= 2(2x)l(o,i](x) + 2[2x - 2(2x - I) ]lu.1](x) for 0 < x ~ t J4x

- \4(1 - x) for

t < x ~ I,

and

for 0 < x <t for for

t < x <i i < x ~ I.

lx/x), fxl x), and lx/x) are sketched in Fig. 3, and an approach to normality can be observed. (In fact, the inflection points of I X 3(X) and of the normal approximation occur at the same points!) We have given the distribution of the sample mean from a uniform distribution on the interval (0, I]; the distribution of the sample mean from a uniform distribution over an arbitrary interval (a, b] can be found by transformation.

3.7 Cauchy Distribution

Let Xl' ... , Xn be a random sample from the Cauchy density
I f(x) = n{J{1 + [(x - a)/Pf} ;

SAMpLiNG FROM THE NORMAL DISTRIBUTIONS

239

l~--~~--------~----

FIGURE 3

~----~-------L----~~~x

then X" has this same Cauchy distribution for any n. That is, the sample mean has the same distribution as one of its components. We are unable to easily verify this result. The moment-generating-function technique fails us since the moment generating function of a Cauchy distribution does not exist. Mathematical induction in conjunction with the convolution formula produces integrations that are apt to be difficult for a nonadvanced calculus student to perform. The result, however, is easily obtained using complex-variable analysis. In fact, if we had defined the characteristic function of a random variable, which is a generalization of a moment generating function, then the above result would follow immediately from the fact that the product of the characteristic functions of independent and identically distributed random variables is the characteristic function of their Sum. A major advantage of characteristic functions over moment generating functions is that they always exist.

..

4 4.1

SAMPLING FROM THE NORMAL DISTRIBUTIONS Role of the Normal Distribution in Statistics

It wi1l be found in the ensuing chapters that the normal distribution plays a very

predominant role in statistics. Of course, the central-limit theorem alone ensures that this win be the case, but there are other almost equally important reasons.

240

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

In the first place, many populations encountered in the course of research in many fields seem to have a normal distribution to a good degree of approximation. It has often been argued that this phenomenon is quite reasonable in view of the central-limit theorem. We may consider the firing of a shot at a target as an i11ustration. The course of the projectile is affected by a great many factors, a11 admittedly with small effect. The net deviation is the net effect of all these factors. Suppose that the effect of each factor is an observation from some population; then the total effect is essential1y the mean of a set of observations from a set of populations. Being of the nature of means, the actual observed deviations might therefore be expected to be approximately normally distributed. We do not intend to imply here that most distributions encountered in practice are normal, for such is not the case at all, but nearly normal distributions are encountered quite frequently. Another consideration which favors the normal distribution is the fact that sampling distributions based on a parent normal distribution are fairly manageable analytically. In making inferences about populations from samples, it is necessary to have the distributions for various functions of the sample observations. The mathematical problem of obtaining these distributions is often easier for samples from a normal population than from any other, and the remaining subsections of this section will be devoted to the problem of fmding the distributions of several different functions of a random sample from a normally distributed population. In applying statistical methods based on the normal distribution, the experimenter must know, at least approximately, the general form of the distribution function which his data follow. If it is normal, he may use the methods directly; if it is not, he may sometimes transform his data so that the transformed observations follow a normal distribution. When the experimenter does not know the form of his population distribution, then he may use other more general but usual1y less powerful methods of analysis called nonparametric methods. Some of these methods wi11 be presented in the final chapter of this book.

4.2

Sample Mean

One of the simplest of all the possible functions of a random sample is the sample mean, and for a random sample from a normal distribution the distribution (exact) of the sample mean is also normal. This result first appeared as a special case of Example 12 in Chap. V. It is repeated here.

SAMPLING FROM THE NORMAL DISTRIBUTlONS

241

Theorem 6 Let Xn denote the sample mean of a random sample of size n from a normal distribution with mean J.1 and variance 0'2. Then Xn has 2 a normal distribution with mean it and variance a /n.
To prove this theorem we shall use the moment-generatingfunction technique.
PROOF

= tG [

tXt] .n exp tXt] =.n tG expn n

n n [

,:1

,~\

=
=

l== I

n
n

mXi

t) =

i= I

1 (at) nexp [J.1t - +- 2] n 2 n

n

exp [ J.1t +

;(0'1)2]

'

which is the moment generating function of a normal distribution with mean J.1 and variance a2 /n. 1//I Since we have the exact distribution of Xn , in considering estimating J.1 with Xn , we will be able to calculate, for instance, the (exact) probability that our .. estimator)) Xn is within any fixed amount of the unknown parameter J.1.

4.3

The Chi..Square Distribution

The normal distribution has two unknown parameters J.1 and 0'2. In the previous subsection we found the distribution of X n , which" estimates" the unknown J.1. In this subsection, we seek the distribution of

which estimates" the unknown 0'2. A density function which plays a central role in the derivation of the distribution of 8 2 is the chi-square distribution.
H

Definition 8 Chi..square distribution If X is a random variable with density

(21)

242

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

then X is defined to have a chi-square distribution with k degrees offreedom; or the density given in Eq. (21) is called a chi-square density with k degrees offreedom, where the parameter k, called the degrees offreedom, is a positive integer. IIII

Remark We note that a chi-square density is a particular case of a gamma density with gamma parameters r and A equal, respectively, to
kl2 and

t. Hence, if a random variable

X has a chi-square distribution,

tS'[X] = kl2 = k,

X] - kl2 - 2k var [ - (1/2)2 ,

(22)

and

mX<t) -

_[_t]k12 _ [1
~ 2
-

- 2t

]k12

'

t < 1/2.

(23)

1111
Theorem 7 If the random variables Xi' i = 1,2, ... , k, are normal1y and
independently distributed with means f.11 and variances
U=
j=l

af, then Ik (X.' _11.)2

Fl

(Ii

has a chi-square distribution with k degrees of freedom. Write Zi = (Xl - f.1i)l(Ii; then Zi has a standard normal distribution. Now
PROOF

mu(t)

= tS'[exp tV] = tS'[exp(t I zf)]

=

,,[;v.exp tZ;]
J2n
2t

lI/[ex ptZn

But

oo

1 e -t(1-2t)z 2 d z

-00

foo
-00

/1-2t J21C
t<

)1 1

e -!(1-2t)Z2 d z
1

/1 - 2t

for

2.'

SAMPLING FROM THE NORMAL DISTRIBUTIONS

243

the ]a tter integral being unity since it represents the area under a norma] curve with variance 1/(1 - 2t). Hence,

i=l

1 = (1 n0"[exp tZlJ = n J 1-2t 1 2t k k

i=l

)k12

for

t<

2:'

the moment generating function of a chi-square distribution with k degrees of freed om. fill

Corollary If Xl, ... , Xn .is a random samp]e from a norma] distribution

with mean J.1 and variance
(12,

then U =

L (Xi i= 1

J.1)2/(12 has a chi-square

distribution with n degrees of freedom.

fill

We might note that if either J.1 or (12 is unknown, the U in the above coro]]ary is not'a statistic. On the other hand, if J.1 is known and (12 is unknown,

we could estimate

,,2 with (I fn)

it.

(X, - li)2 (note that .c

[(I In),t,

(X, - JI)2]

=
of

(lln),t. C[(X, - JI)2] = (1/n),t,

(1/n)

,,2 = ,,2),

and

find

the

distribution

L (Xi i= 1

J.1)2 by using the coro]]ary.

Remark In words, Theorem 7 says, "the sum of the squares of independent standard normal random variables has a chi-square distribution with degrees of freedom equa] to the number of terms in the sum." fill

Theorem 8 If Zl' Z2' ... , Zn is a random samp]e from a standard

normal distribution, then:
(i)
(i~)

Z has a norma] distribution with mean Qand variance lin.

Z and
n

L (Zi i= 1

2)2 are independent.

(iii)

L (Zi i= 1

Z)2 has a chi-square distribution with n - I degrees

of freedom. Theorem 6.
PROOF "-

(Our proof wi]] be incomp]ete.) (i) is a special case of We wi11 prove (ii) for the case n = 2. If n = 2,

Z=

Zl

+ Z2
2

244

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

and

L(Z, - 2)2 (Z. _Z. ;Z2)' + (Z2 _Z. ; z2f

=

=
-

(ZI - Z2)2

(Z2 - ZI)2

+ '

(Z2 - ZI)2 .

so 2 is a function of ZI + Z2, and L (Zi - 2)2 is a function of Z2 - ZI; so to prove 2 and L (Zj - Z)2 are independent, it suffices to show that ZI + Z2 and Z2 - ZI are independent. Now

and, similarly,
mz Z -Z 1(t 2 )

= exp t~.

Also,
mZ +zz, ZZ-Zl(t It t 2 ) 1

= G[e (Zl +zz)+tz(Zz-Zt)] = G[e(h -tz) Zl e(tl +tz)ZZ] =

1

G[e(tt-t z )zJ]G[e(t l +t 1)Z2]

= et(t1-tZ)Zetch +tZ)2 =

exp

tf exp t~

= mZ 1 +Z z(tl)mZ z -Z t (t2);

and since the joint moment generating function factors into the produc~ of the marginal moment generating functions, ZI + Z2 and Z2 - ZI are independent. To prove (iii), we accept the independence of Z and

L (Zj 1

2}2 for

arbitrary n. Let us note that L zf = L (Z, - 2 + Zf = L (Zl - 2)2 + 22 L(Zj - Z) + L Z2 = L(Zj - 2)2 + nZ2; also L(Zi - zf and nZ2 are independent; hence

So,
t < 1/2

SAMPLING FROM THE NOR.MAL DISTRIBUTIONS

245

noting that J~Z has a standard normal distribution implying that nZ/ has a chi-square distribution with one degree of freedom. We have shown that the moment generating function of L (Zi - Z)2 is that of a chi-square distribution with n - 1 degrees of freedom, which completes
~~~

Theorem 8 was stated for a random sample from a standard normal distribution, whereas if we wish to make inferences about 11 and a 2 , our sample is from a normal distribution with mean p and variance a 2 Let Xl' ... , Xn denote the sample from the norma] distribution with mean Il and variance a 2 ; then the Zi of Theorem 8 could be taken equal to (Xi - II)la. (i) of Theorem 8 becomes: (i') Z = (lIn) L (X, - Il)la = (X - tt)/a has a normal distribution with mean 0 and variance 1In.
(H)

of Theorem 8 becomes: (ii') Z = (X - p)la and L(Zi - Z)2 = L [(Xi - p)la - (X - p)laf = L [(Xi - %)2/a2] are independent, which implies X and L (Xi - %)2 are independent.

(iii) of Theorem 8 becomes: (iii') L (Zi - Z)2 = L [(Xi - X)2/a 2 ] has a chi-square distribution with n - 1 degrees of freedom.
n

Corollary If 8 2

[l/(n - 1)]

i= I

(Xi - X)2 is the sample variance of a

(r2,

random sample from a normal distribution with mean Il and variance then
",

(24) has a chi-square distribution with n - 1 degrees of freedom.

PROOF

This is just (iii').

IIII

Remark Since 8 2 is a linear function of U in Eq. (24), the density of 8 2 can b~ obtained from the density of U. It is
f~p(y)

= ( n - 2 1)(n-1)/2
2a

r[(n _ 1)/2]

. y(n-3)/2 e -(n-l)y/2a j
2

(y)
(0. cO)

(25)

IIII

246

sAMPLING AND SAMPLING DISTRIBUTIONS

VI

Remark The phrase "degrees of freedom" can refer to the number of

independent squares in the sum. For example, the Sum of Theorem 7 has k independent squares, but the sum in (iii) of Theorem 8 has only n - I independent terms since the relation L (Zi - Z) = 0 enables one to compute anyone of the deviations Zj - Z, given the other n - I of them. IIII All the results of this section apply only to normal populations. In fact, it can be proved that for no other distributions (i) are the sample mean and sample variance independently distributed or (ii) is the sample mean exactly normally distributed.

4.4

The F Distribution

A distribution, the F distribution, which we shall later find to be of considerable practical interest, is the distribution of the ratio of two independent chi-square random variables divided by their respective degrees of freedom. We suppose that U and V are independently distributed with chi-square distributions with m and n degrees of freedom, respectively. Their joint density is then [see Eq. (21)]
_ 1 (m-2)/2 (n-2)/2 -!(u+v) Iv. v(u, v) - r(m/2)r(n/2)2(m+n)/2 u v e 1(0, oo)(u)/(o. oo)(v). (26)

We shall find the distribution of the quantity

X- Ulm - Vln'
(27)

which is sometimes referred to as the variance ratio. To find the dIstribution of X, we make the transformation X = (Ulm)/(Vln) and Y = V, obtain the joint distribution of X and Y, and then get the marginal distribution of X by integrating out the y variable. The Jacobian of the transformation is (mln)y; so
r
JX, y

) _ m 1 (m x )(m-2)/2 (n-2)/2 e -H(m/n)xy+y] x, Y - n y r(m/2)r(n/2)2(m+n)/2 n y y ,

and
00

fx(x) = fix, y(x, y) dy

o

_ 1 (m)m/2 x(m-2)/2 feCI (m+n-2)/2 e -H(m/n)x+ IJy dy - r(m/2)r(n/2)2(m+n)/2 n 0 y

r[(m + n)/2] (m)m/2 = r(m/2)r(n/2) ~ [1 x(m-2)/2 + (m/n)x](m+n)/2 1(0,. eCllx ). (28)

SAMPLING FROM THE NORMAL DISTRIBUTIONS

247

Definition 9 F distribution If X is a random variable having density given by Eq. (28), then X is defined to be an F-distributed random variable with degrees offreedom m and n. jjjj
The order in which the degrees of freedom are given is important since the density of the F distribution is not symmetrical in m and n. The number of degrees of freedom of the numerator of the ratio mjn that appears in Eq. (28) is always quoted first. Or if the F-distributed random variable is a ratio of two independent chi-square-distributed random variables divided by their respective degrees of freedom, as in the derivation above, then the degrees of freedom of the chi-square random variable that appears in the numerator are always quoted first. We have proved the following theorem.

Theorem 9 Let U be a chi-square random variable with m degrees of freedom; let V be a chi-square random variable with n degrees of freedom, and let U and V be independent. Then the random variable
X= Ujm Vjn

is distributed as an F distribution with m and n degrees of freedom. density of X is given in Eq. (28).

The
jjjj

The following corollary shows how the result of Theorem 9 can be useful in sampling.

Corollary If Xl' ... , X m + 1 is a random sample of size m + I from a normal population with mean I1x and variance a 2 , if Y1 , .. , Yn + 1 is a random sample of size n + I from a normal population with mean Jiy and variance a 2 , and if the two samples are independent, then it follows
m+l

that (lj(

L
1

(Xi - X)2 is chi-square distributed with m degrees of

n+l
1

freedom, and (1 j( 2)

L (Yj -

y)2 is chi-square-distributed with n degrees

of freedom; so that the statistic

L (Xi L (Yj has an F distribution with m and n degrees of freedom.

X)2jm
y)2jn

jjjj

248

sAMPLING AND SAMPLING DISTRlBunoNS

VI

We close this subsection with several further remarks about the F distribution.

Remark If X is an F-distributed random variable with m and n degrees of freedom, then

$[X] =

n-2

for n > 2

and
2n2(m + n - 2) var [ X ] = ---::---..;..men - 2)2(n - 4)
PROOF

fer n > 4.

(29)

At first it might be surprising that the mean depends only on the degrees of freedom of the denominator. Write X as in Eq. (27); that is,

X
then
$[X]

Ulm.
Vln'

= $ [-Ulm]
Vln

n =-$[U]$ m

[1]
V

But $[U] = m by Eq. (22), and

$

[~] =
V

1
1

r(n/2) 2 r(n/2)

(!)TI/2 J"oo ! . v(TI-2)/2e-tv dv

0

(~)TlI2

= r[(n and so
$[X]

Joov(n-4)/2 e-tv dv 2 2 2)/2] (!)nI2(!) -(,,-2)/ =

0

1
n- 2

r(n/2)

n) = (m $[U]$

[1] =
V

m n - 2 = n - 2.

The variance formula is similarly derived.

1II1

Remark If X has an F distribution with m and n degrees of freedom, then 11 X has an F distribution with nand m degrees of freedom. This upper tail only. For result allows one to table the F distribution for the ,

SAMPLING FROM THE NORMAL DISTRIBUTIONS

249

example, if the quantile '.95 is given for an F distribution with m and n degrees of freedom, then the quantile '~05 for an F distribution with nand m degrees of freedom is given by 1/'.95' In general, if X has an F distribution with m and n degrees of freedom and Y has an F distribution with nand m degrees of freedom, then the pth quantile point of X, is the reciprocal of the (1 - p)th quantile point of Y, ,'l-P' as the following shows: .

'p,

but
1 - p = P[ Y < l;~ - p] ;

so
I

'l-P =

'p'

1/11

Remark If X is an F-distributed random variable with m and n degrees

of freedom, then

W=

mXln 1 + mXln

has a beta density with parameters a = m/2 and b = n/2.

//1/

4.5

Student's t Distribution

Another distribution of considerable practical importance is that of the ratio of a standard normally distributed random variable to the square root of an independently distributed chi.square random variable divided by its degrees of freedom. That is, if Z has a standard normal distribution, if U has a chisquare distribution with k degrees of freedom, and if Z and U are independent, we seek the distribution of
X=

jUlk

.....

The joint density of Z and U is given by

(30)

,."

250

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

If we make the transformation X =

zljuii

and Y = U) the Jacobian is

jYlk, and so
Ix. y(x, y) _

If
00

1 1 (l)kf2 ~ __ _ (kf2)-1 -'t.v -txly/k 'k j2n r(k/2) 2 y e e /(o.oo)(Y)

Ix(x) = fIx. y(X, y) dy

-00

=
-

fon r(!/2) (~r/2 foro

r[(k + 1)/2] 1 r(k/2) jkn (1

y'12-1+te-t(1+X'I')Y dy
(31)

1 + x 2 /k)(k+ 1)/2'

Definition 10 Student's t distribution If X is a random variable having density given by Eq. (31), then X is defined to have a Student's t distribution, or the density given in Eq. (31) is called a Student's t distribution with k degrees offreedom. IIII We have derived the following result. Theorem 10 If Z has a standard normal distribution, if U has a chisquare distribution with k degrees of freedom, and if Z and U are independent, then Z/j Ulk has a Student's t distribution with k degrees of freedom. IIII The following corollary shows how the result of Theorem lOis applicable to sampling from a normal popUlation. Corollary If Xl' ... , Xn is a random sample from a norm~l distribution with mean fl and variance 0'2, then Z = (X - fl)/(O'IJn) has a standard normal distribution and U = (1/0'2) (Xi - xf has a chi-square distribution with n - 1 degrees of freedom. Furthermore, Z and U are independent.(see Theorem 8); hence

(X - fl)/(O'/j~)

_ jn(n - I)(X - fl)

j(1/O' 2 )

(Xi - Xf/(n - I)

j I (Xi -

1)2
/ /11

has a Student's t distribution with n - 1 degrees of freedom.

ORDER STATISTICS

251

We might note that for one degree of freedom the Student's t distribution reduces to a Cauchy distribution; and as the number of degrees of freedom increases, the Student's t distribution approaches the standard normal distribution. Also, the square of a Student's t-distributed random variable with k degrees of freedom has an F distribution with I and k degrees of freedom. Remark If X is a random variable having a Student's t distribution with k degrees of freedom, then
C[X]

=0
PROOF

if k > 1

and

var [X] '= kl(k - 2)

if k > 2.

(32)

The first two moments of X can be found by writing

X = ZIJUlk as in Theorem 10 and using the independence of Z and U. The actual derivation is left as an exercise. IIII
This completes Sec. 4 on sampling from the normal distribution. Note that we considered the distribution of functions of only two different statistics, namely, the sample mean and sample variance. In the next chapter we will find that these two statistics are the only ones of interest in sampling from a normal distribution; they will turn out to be sufficient statistics.

ORDER STATISTICS

5.1

Definition and Distributions

In Subsec. 2.4 we defined what we meant by statistic and then gave the sample moments as examples of easy-to-understand statistics. In this section the concept of order statistics will be defined, and some of their properties will be investigated. Order statistics, like sample. moments, play an important role in statistical inference. Order statistics are to population quantiles as sample moments are to population moments. Definition 11 Order statistics Let Xl' X 2 , , Xn denote a random sample of size n from a cumulative distribution function F(')' Then Yt < Y2 < ... < Yn , where Y i are the Xi arranged in order of increasing magnitudes and are defined to be the order statistics corresponding to the random sample Xl' .. " Xn IIII

252

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

We note that the Yj are statistics (they are functions of the random sample Xh X 2 , , Xn) and are in order. Unlike the random sample itself, the order statistics are clearly not independent, for if Y j > y, then Yj + l > y. We seek the distribution, both marginal and joint, of the order statistics. We have already found the marginal distributions of YI = min [Xl' ... , Xn] and Yn = max [Xl' ... , Xn] in Chap. V. Now we will find the marginal cumulative distribution of an arbitrary order statistic.

Theorem 11 Let YI < Y 2 < .. ~ Yn represent the order statistics from a cumulative distribution function F( . ). The marginal cumulative distribution function of Y a , a = 1, 2, ... , n, is given by
F Yr.,(y)
PROOF

)=tZ

(~)[F(y)]j[l
J

- F(y)r- j

(33)

For fixed y, let

then

2: Zi = =1

the number of Xi < y.

Note that Now

2: Zj has a binomial distribution with parameters nand F(y).

i= I

FyJy) = P[Ya < y] = P[2: Zi > a] =

}=tZ

(~)[F(y)]j[l J

F(y)r- i .

The key step in the proof is the equivalence of the two events {Ya < y} and {2: Zj > a}. If the ath order statistic is less than or equal to y, then surely the number of Xi less than or equal to y is greater than or equal to a, and conversely. 1111

Corollary
and
Fy,(Y) =

Fy"(y) =

J=n

.f (n.) [F(y)J1[l J

j F(y)r- = [F(Y)r,

J.

C)[F(y)]i[l - F(Y)r- i = 1 - [1 - F(y)]".

/III

ORDER STATISTICS

253

Theorem 11 gives the marginal distribution of an individual order statistic in terms of the cumulative distribution function F(')' For the remainder of this subsection, we will assume that our random sample Xl' ... , Xn came from a probability density function f('); that is, we assume that the random variables Xi are continuous. We seek the density of Y a , which, of course, could be obtained from Eq. (33) by differentiation of Fy"'(y). Note that

fy,/y)
=

lim Fy,.{y
A,.-O

+ Ay) Ay

Fy",(Y)

lim P[y < Ya y A,.-O Ay

+ Ay]

= lim P[(a: -l)ofthe Xi ~ y; one Xi in(y,y + Ay];(n - a:) of the Xi> y + Ay]
A,.-O

Ay

= lim
A,.-O

n! [F(y)r-l[F(y \( a: - 1) U !(n - a:)!

+ Ay) -

F(y)][1 - F(y Ay

+ Ay)]n-a}

- (a: _ l)!('n _ a:)! [F(y)]a-l[1 - F(y)]n-af(y)

n'

We have made sensible use of the multinomial distribution. derive the joint density of Ya and Yp for 1 < a: < p n.

Similarly, we can

fy,u y,(x, y) Ax Ay ~ P[x < Ya < X + Ax; y < Yp < y + Ay] ~ P[(a: - 1) of the Xi < x; one Xi in (x, x + Ax];
(P-a:-1)ofthe Xi in (x+Ax,y]; one Xi in (y,y

+ Ay]; (n - P) of the Xi > Y + Ay]

~----~-----------------

nl (a: - I)! 1 t(P - a: - I)! l!(n - {I)!

x [F(x)]a-l[F(y) - F(x
hence

+ Ax)]fl-a-l[1

- F(y

+ Ay)]n-Pf(x) Axf(y) Ay;

fy",. Y,(X' y)

(a: - 1)1 (P

n' _ a: '_ I)! (n _

P)![F(x)]a-l[F(Y) - F(x)]p-a-l[l - F(y)]n-Pf(x)f(y)

for x < y, and

fYt%. yiX, y)
=

for x >y.

254

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

In general, fy 10''', y,,(y l' . . . ,YJ

= lim
Ayr-+O

nAYi
n

P[YI < Y 1 < Yl

+ AYl; ... ; Yn

< Y n < Yn

+ AYn] + AYn]]

i= 1

= lim
AYt-+ O

i =1

n
n n

1
AYI

P[one Xi in (y., Yl + AYl]; . " ; one Xi in (Yn, Yn

lim
AYt-+O

n =
1

n!
AYI

[F(y 1 + Ay 1) - F(y 1)] .... [F(Yn

+ AYn) -

F(Yn)]

for Yl < Y2 < ... < Yn'

andfYl, ... , Yn(Yl"'" Yn) = 0, otherwise. We have derived the following theorem.

Theorem 12 Let Xl' X 2 , , Xn be a random sample from the probability density function f(') with cumulative distribution function F(). Let Y1 < Y2 < ... < Yn denote the corresponding order statistics; then
fyJy) = (a _ 1)! en

n' _ a)! [F(y)]'~-I[1

-

- F(y)]n-'i(Y);

(34)

fy"" YfJ(x, y) = '(a - l)! (P

n! a - 1)! (n - P) !

x [F(x)]a-l[F(y) - F(x)]p-a-l x [1 - F(y)]n-Pf(x)f(y)I(x. oo)(Y);

(35)
(36)

fYI .... Yn (Yl, ... , Yn)

{~!fly,)

..... f(y.)

for Yl < Y2 < . . . < Yn otherwise.

IIII

Any set of marginal densities can be obtained from the joint density fyIo .... Yn(Yl' ... , Yn) by simply integrating out the unwanted variables.

5.2

Distribution of Functions of Order Statistics

In the previous subsection we derived the joint and marginal distributions of the order statistics themselves. In this subsection we will find the distribution of certain functions of the order statistics. One possible function of the order statistics is their arithmetic mean, equal to 1 n - L Yj
n
j= 1

ORDER

STATISTICS 255

Note, however, th~t (lIn)

II

Yj

(lin)

L Xi'
i=l

II

the sample mean, which was the

j=l"

subject of Sec. 3 of this chapter, order statistics,

We define now some other functions of the

Definition 12 Sample median, sample range, sample midrange Let Yl < ... ~ Yn denote the order statistics of a random sample Xl' " " Xn from a density f( . ), The sample median is defined to be the middle order statistic if n is odd and the average of the middle two order statistics if n is eye..!!, The sample range is defined to be Yn - Yl , and the sample midIIII range is defined to be (Y} + Yn)/2, . IS ,~t'3 t., Id , If the sample SIze odd, then t h e d'Istn'b' ution 0 f t h e samp e me ' Ian IS
I

given by Eq. (34); for example, if n = 2k + I, where k is some positive integer, then" Yk + l is the sample median whose distribution is given by Eq, (34). If the sample size is even, say n = 2k, then the sample median is (Yk + Yk + l )/2, the distribution of which can be obtained by a transformation starting with the joint density of Yk and Yk + l , which is given by Eq. (35), We derive now the joint distribution of the sample range and midrange, from which the marginals can be obtained. By Eq, (35), we have fYl. y .(x, y) = n(n - 1)[F(y) - F(x)]n-2f(x)f(y) for x < y, (37) Make the transformation R = YII - Yl and T = (Yl + Yn )/2 or r = y - x and t = (x + Y)/2. Now x = t - r12, and y = t + r12; hence ax ax ar at ay ar and we obtain Theorem 13. Theorem 13 If R is the sample range and T the sample midrange from a probability density function, then their joint distribution is given by fR, r(r, t) = n(n - 1)[F(t
-

J=

ay at

-! !

1 1

-1,

+ rI2)- F(t -

rI2)]n-2f(t - rI2)f(t
,

+ r12)

for r > 0,

(38)

and the marginal distributions are given by and fT(t)

=

foo
o

fR, T(r, t) dr.

(39)

IIII

256

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

EXAMPLE 6

Let Xb ... , Xn be a random sample from a uniform distribuHere Jl is the mean, and
1

tion on (Jl - j3a, Jl + j3a). of the sampled population.

a2 is the variance

I(x) =

2)30' 1(p._..,I3(1, p.+..,I3(1)(X),

J

(40)

JR(r) = ffR. T(r, t) dt = n(J- 1) (2 3ay

,.-2 (2J3U -

r)l(o 2,130)(r).

(41)

We note that IR(r) is independent of the parametero Jl.

fT(t) = ffR. r(r, t) dr

n(n - 1) = (2j30't

min[2t-2(p.-..,I30'),2(p.+..,I3(1)-2t]

rn -

dr . I

(p.-..,I30'.p.+..,I3(1)

- (t)

which simplifies to
Jr(t) =

2:/Ju C J 3: + 1r'I(-I.o{/3:) + 2j3U (1 - tJ3 1[0'I)C3:)'

:r'

(42)

From Eq. (41), we can derive 8[R]

= 2j3a(n - l)/(n + 1).

/1//

Certain functions of the order statistics are again statistics and may be used to make statistical inferences. For example, both the sample median and the midrange can be used to estimate Jl, the mean of the population. For the uniform density given in the above example, the variances of the sample mean, the sample median, and the sample midrange are compared in Problem 33.

5.3

Asymptotic Distributions

In Subsec. 3.3, we discussed the asymptotic distribution of the sample mean X n. We saw that Xn was asymptotically normally distributed with mean Jl and variance a2 /n. We now consider the question: Is there an asymptotic distribution for the sample median? We will state (without proof) a more general result.

ORDER STATISTICS

257

Since for asymptotic results the sample size n increases, we let yin) < Y!IJ) < ... < y!IJ) denote the order statistics for a sample of size n. The superscript denotes the sample size. We will give the asymptotic distribution of that order statistic which is approximately the (np)th order statistic for a sample of size n for any 0 <p < I. We say "approximately" the (np)th order statistic because np may not be an integer. Define Pn to be such that nPn is an integer and Pn is approximately equal to P; then Y!;~ is the (nPn)th order statistic for a sample of size n. (If Xl' ... , Xn are independent for each positive integer n, we will say Xl' ... , X n , ... are independent.) Theorem 14 Let Xl' ... , Xn , be independent identically distributed random variables with common probability density f(') and cumulative distribution function F(). Assume that F(x) is strictly monotone for 0< F(x) < 1. Let p be the unique solution in x of F(x) = P for some 0< P < 1. ~ep is the pth quantile.) Let Pn be such that nPn is an integer and nlPn - pi is bounded. Finally, let Y;;~ denote the (npn)th order statistic for a random sample of size n. Then Yn~~ is asymptotically distributed as a normal distribution with mean e,p and vanance p(I - p)ln[f(ep)p. IIII

EXAMPLE 7 Let P = -!; then e p is the population median, and Theorem 14 states that the sample median is asymptotically distributed as a normal distribution with mean the popUlation median and variance 1/4n[f(e,1/2)]2. In particular, if f( .) is a normal density with mean fl and variance a 2 , then the sample median is asymptotically normally distributed with mean fl and variance 1/4n[f(fl)f = TCa 2 /2n. Recall that the sample mean is normally distributed with mean fl and variance (J2In. 11/1 In Theorem 14 above we considered a certain kind of asymptotic distribution of order statistics. We will now consider yet another kind. In the above we looked at the asymptotic distribution of that order statistic which was approximately the (np)th order statistic for a sample of size n. Such an order statistic had (approximately) lOOp percent of the n observations to its left. That is, its relative position remained unchanged as n, the sample size, increased; it always had (approximately) the same percentage of the n observations to its left. We will now consider the asymptotic distribution of that order statistic whose absolute position remains unchanged. That is, we consider the asymptotic distribution of, say, Yin) for fixed k and increasing n. yt) is the kth smallest order statistic for a sample size n > k, and k remains fixed. In order to make the presentation somewhat simpler, we will take k = 1,

.258

SAMPUNO AND SANPUNO DISTRIBUTIONS

VI

in which case Y1 n ) is the smallest of the n observations. We note that we could just as well consider the kth largest order statistic, namely y~=:?k+" which for k = 1 specializes to y~n), the largest order statistic for a sample of size n. Either y~n) or y~n) is often referred to as an extreme-value statistic. Practical applications of extreme-value statistics are many. The old adage that a chain is no stronger than its weakest link provides a simple example. If Xi denotes the" strength" of the ith link of a chain with n similar links, then y~n) = min [Xb "', X,.] is the" strength" of the chain. Also, in measuring the results of certain physical phenomena such as floods, droughts, earthquakes, winds, temperatures, etc., it can be seen that under certain circumstances one is more interested in extreme values than in average values. For instance, it is the extreme earthquake or flood, and not the average earthquake or flood, that is more damaging. We can see that results, whether exact or asymptotic, for extreme-value statistics Can be just as important as results for averages. For the most part we will concentrate on finding the asymptotic distribution of y~,.). One might wonder why we should be interested in an asymptotic distribution of y~n) when the exact distribution, which is given by Fyn(n)(Y) = [F(y)]n, where F(') is the c.d.f. sampled from, is known. The hope is that we will find an asymptotic distribution which does not depend on the sampled c.d.f. F(')' We recall that the central-limit theorem gave an asymptotic distribution for Xn which did not depend on the sampled distribution even though the exact distribution of X,. could be found. In searchi~g for the asymptotic distribution of y!n), let us pattern our development after what was done in deriving the asymptotic distribution of Xn According to the law oflarge numbers, Xn has a degenerate limiting distribution; that is, the limiting c.d.f. of Xn is the cumulative distribution that assigns all its maSS to the point /1. Such a limiting distribution is not useful if one intends to use the .limiting distribution to approximate probabilities of events since it assigns each event a probability of either 0 or 1. To circumvent such difficulty, we first" centered" the values of Xn by subtracting /1, and then we " inflated" the values of Xn - /1 by mUltiplying them by J~IO', and, consequently, we were able to get a non degenerate limiting distribution; that is, accordingtothecentrallimit theorem, J~( Xn - /1)/0' had a standard normal distribution as its limiting distribution. A general procedure, when one is looking for a limiting distribution of, say, Z,. , is to first'" center" the Zn by subtracting a constant, sayan, and to then" scale" Zn - an by dividing by another constant, say bn . In the case of the central-limit theorem, Zn = Xn, an /1, and bn = O'/J~. In the case of Theorem 14 above, Zn = Y~;~, an e p, and bn = p(l - p)/n[f(ep)f. Por both of these two Cases the sequence of constants {an} did not depend on n. In the Case at hand, namely when Zn = y!n), the sequence of constants {on} is likely to

ORDER STATISTICS

259

depend on n since y!n) tends to increase with n. examples.

Let us look at a couple of

EXAMPLE 8 Consider sampling from the logistic distribution; that is, F(x) = (1 + e-~-l. Find the limiting distribution of (y~n) - an)/bn . There are two problems: First, what should we take the sequences of constants {an} and {bn} to be? And, second, what is the limiting distribu~ tion of (y~n) - an)/bn for the selected constants {an} and {bn}? It seems reasonable that the "centering" constants {an} should be close to c[y~n)]; so we seek an approximation to C[ y~n)]. Now F(Xd, ... , F(Xn) is a random sample from the uniform distribution over (0, 1); hence F( y~n) is the largest of a sample of size n from a uniform distribution over (0, 1). ThatC[F(y~n)] = n/(n + 1) can then be routinely derived. Now F(c[y~n)]) ~ C[F( y~n)] or
F(c[y~n)]) =

+ exp (_c[y~n)])}-1 = 1 - {l + exp (C[ y~n)])}-l

{l
~

1
n

-en + 1)-1
n

---

+1

= C[F( y~n)],
which implies that

n
or that

exp (C[ y~n)]

C[ y~n)] ~ loge n.

Finally, since C[ y~n)] ~ log n (from here on we use log n for loge n), a reasonable choice for the sequence of "centering" constants {an} seems to be the sequence {log n}. Weare seeking

limp[y~n) n-+oo

an <

Y] = 1imp[y~n) -log n < Y]

n-+oo

bn

bn

= 1imP[Y~") < bny + log n]

n-+oo

lim [F(bny
n-+oo n-+oo

+ log n)]n

= lim(l +
= lim(}
n-+oo

e-bnY-logn)-n

+ (l/n)e-bny)-n

260

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

Hence, if {an} and {bn} are selected so that {an} = {log n} and {bn} = {l}, respectively, then the limiting distribution of (y~n) - an)lbn = y~n) - log n is exp ( - e - Y). IIII

EXAMPLE 9 Consider sampling from the exponential distribution so that F(x) = (l - e-).x)I(o, oo)(x). Again, let us find the limiting distribution of (y~n) _ an)lbn . As in Example 8, tC[F( y~n] = nl(n + 1) = 1 -l/(n + 1). Now

and

so

n + I ~ exp { - AtC[ y~n)]},

or

] tC[ y!n)] ~ J log (n

+ 1) ~ J log n.

Hence, it seems reasonable to use an = (IIA) log n.

Y]

= limp[Yn n ..... oo

] ! log n < b Y A

= !~n;, [F(b n Y+ ~ log n)

= lim(l e-;'bny-logn)"

' (1 - 1 e-;'b = lim

,. ..... 00

l1

)n

Hence the limiting distribution of(y!n) - an)lbn = [y~n) - (1IA) log n]/O/A) is exp (-e- Y ). We note that we obtained the same limiting distribution here as in Example 8. Here we were sampling from an exponential distribution, and there we were sampling from a logistic distribution.

IIII

ORDER STATISTICS

261

In each of the above two examples we were able to obtain the limiting distribution of (y~n) - Qn)lbn by using the exact distribution of y!n) and ordinary algebraic manipulation. There are some rather powerful theoretical results concerning extreme-value statistics that tell us, among other things, what limiting distributions we can expect. We can only sketch such results here. The interested reader is referred to Refs. 13, 30, and 35. Theorem 15 Let X., ... , X n , be independent and identically distributed random variables with c.dJ. F('). If (y!n) - an)lbn has a limiting distribution, then that limiting distribution must be one of the following three types:
Gt(y; y)
=

e- Y - Y /(0, oo)(Y),

where y > O.

G2 (y; y) = e-1Y1Y/(-OO,O)(Y)

+ /[0, oo)(Y),

where y > O.

G3(Y) = ex P ( - e - Y).

1111

Theorem 15 states what types of limiting distributions can be expected. The following theorem gives conditions on the sampled F(') that enable us to determine which of the three types of limiting distributions correspond to the sampled F(). Theorem 16 Let X., ... , Xn , be independent and identically distributed random variables with c.d.f. F(). Assume that (y~n) - an)lbn has a limiting distribution. The limiting distribution is: (i)
G1(. ; y) if and only if

. 1 - F(x) 1 1m x-+oo 1 - F( rx)

(ii)

= rY

for every r > O.

G2 (; y) if and only if there exists an Xo such that

and and

F(xo - E) < 1

for every E > O.

1 - F(xo - rx) o<x-+o I - F(xo - x)

lim

= rY

for every r > O.

(iii)

G 3 () if and only if

n-+oo

lim

n[I - F(Pnx

+ an)] = e-

for each x,

262

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

where
Ci" = inf {z:
n - 1

< F(z)

and

Pn = inf {z:

1 - (ne)-l < F(a" + z)}.

fill

Note that if F(') is strictly monotone and continuous, then an is given by F(an) = (n - 1)/n, or an = F-1({n - 1}/n); and P" is given by F(a" + P,,) = 1 (ne)-I, or P" = F- 1(1 - {ne} -1) - a" = F- 1(1 - {ne} -1) - F- 1({n - I}/n),

EXAMPLE 10 Take F(x) = (1 - e-1j/(o. oo)(x) as in Example 9. a" is such that F(a,,) = (n - 1)/n or 1 - e- lrzn = 1 - l/n, which implies that a" = (1/l) log n. p" is such that F(Ci" + P,,) = I - (ne) -lor 1 _ e-1(1/1) log n-1Pn = 1 - (ne)-l, or p" = I/l. for each x, so, as we saw in Example 9, the exponential distribution has G 3 (') as its fill corresponding limiting extreme-value distribution,

EXAMPLE 11 Take F(x) = F(x; y) = [1 - (l - x)yl/(o. 1 )(x) + /[1. oolx). Note that for Xo = I, F(xo) = 1 and F(xo - e) < 1 for every e > O. Also,
, 1 - F(xo - 1'x) I1m o<x"'o 1 F(xo - x)

I'

1m

O<X"'O

(1 - Xo + 1'x)Y (1- Xo + xY

=1">

"

so F('; y) has G 2(; y) as its limiting extreme-value distribution.

fill

EXAMPLE 12 Take F(x) = F(x; y) the c.d.f. of a t distribution with y degrees of freedom.
. 1 - F(x) _ I' f(x) . [1 + (1'X)2/y](Y+ 0/2 hm - 1m = 11m 2 (1'+ 1)/ 2 x"'oo 1 - F(rx) X"'oo 1'f(1'x) x"'oo 1'(1 + x /y)
_
-

y. l' ,

so the t distribution with y degrees of freedom has GI (' ; y) as its limiting fill extreme-value distribution.

ORDER STATISTICS

263

Theorem 16 gives conditions on the sampled c.d.f. F(') that enable us to determine the proper limiting extreme-value distribution for (y~n) - an)jbn The theorem does not tell us what the constants {an} and {b n } should be. If, however, the conditions for the third type are satisfied, then we have
n[l - F(Pnx

+ an)] ~ e- x , as n ~ 00,

and
p

[Y!'~: a. < x]

--+

exp (-e-,,).

Now, p[(y~n) - a,.)jbn <x]

[F(bnx

+ an))"; hence [F(bnx + an)]n ~ exp (-e X),

or

or n[l - F(bnx

+ a,.)] -+ e- x ;

and we see that a,. can be taken equal to ctn and bn = Pn . Thus, for the third type the constants {an} and {bn} are actually determined by the condition for that type. We shall see below that for certain practical applications it is possible to estimate {an} and {bn}. Since the types G1(. ; y) and G 2 (' ; y) both contain a parameter, it can be surmised that the third type G 3(') is more convenient than the other two in applications. Also, G 3 (y) = exp (-e-)') is the correct limiting extreme-value distribution for a number of families of distributions. We saw that it was correct for the logistic and exponential distributions in Examples 8 and 9; it is also correct for the gamma and normal distributions. What is often done in practice is to assume that the sampled distribution F(') is such that exp (-e- Y) is the proper limiting extreme-value distribution; one can do this without assuming exactly which parametric family the sampled distribution F(') belongs to. One then knows that p[(y~n) - an)jbn ~ y] ~ exp (-e-y) for every y as n ~ 00. Hence,

for large fixed n. or

Or

264

SAMPLING AND SAMPLING DISTRIBUTIONS

VI

It is true that

and bn are given in terms of the (1 - 1/n)th quantile and the (1 - 1/ne)th quantile of the sampled distribution; however, for certain applications they can be estimated, in which case we would have an approximate distribution for y!n), a distribution that is valid for a variety of different distributions that could be sampled from. (One might note that in applications of the centrallimit theorem, which states that Xn is approximately distributed as N(Jl, (J2/n), often Jl and (J2 are unknown and consequently they also have to be estimated.) The preceding indicates how powerful the asymptotic extreme-value theory can be. We have merely introduced the subject. For instance, we stated some results for the asymptotic distribution of y~n); one could state similar results for y~n), y1 n), or y~~k+l' The interested reader is referred to Refs. 13, 30, and 35.
On

5.4

Sample Cumulative Distribution Function

We have repeatedly stated in this chapter that our purpose in sampling from some distribution was to make inferences about the sampled distribution, or population, which was assumed to be at least partly unknown. One question that might be posed is: Why not estimate the unknown distribution itself? The answer is that we can estimate the unknown cumulative distribution function using the sample, or empirical, cumulative distribution function, which is a function of the order statistics.

Definition 13 Sample cumulative distribution function Let Xl, X 2 , , Xn denote a random sample from a cumulative distribution function F('), and let y. < Y2 < ... < Yn denote the corresponding order statistics. The sample cumulative distribution function, denoted by Fn(x), is defined by Fix) = (lIn) x (number of Yj less than or equal to x) or, equivalently, IIII by Fn(x) = (lIn) x (number of Xi less than or equal to x).
For fixed x, Fn(x) is a statistic since it is a function of the sample. (The dependence of Fn(x) on the sample may not be clear from the notation itself.) We shall see that Fix) has the same distribution as that of the sample mean of a Bernoulli distribution.

Theorem 17 Let FnCx) denote the sanlple cumulative distribution function of a random sample of size n from F( . ); then
P [F.(X)

=~]

=(:)

[F(x)l'[l - F(x)r',

k = 0, 1, ... , n.

(43)

PROBLEMS

265

PROOF

Let Zi = Ic- rCJ,X](Xi ); then Zi has a Bernoulli distribution

with parameter F(x). Hence,

n

L Zi' 1

which is the number of Xi less than

or equal to x, has a binomial distribution with parameters nand F(x). But

Fn(x) =

(lIn) L Zi .
1

The result follows.

IIII

Much more could be said about the sample cumulative distribution function, but we will wait until Chap. Xl on nonparametric statistics to do so.

PROBLEMS
1

Give an example where the target population and the sampled population are the same. (b) Give an example where the target population and the sampled population are not the same. (a) A company manufactures transistors in three different plants A, B, and C whose manufacturing methods are very similar. It is decided to inspect thOse transistors that are manufactured in plant A since plant A is the largest plant and statisticians are available there. In order to inspect a week s production, 100 transistors will be selected at random and tested for defects. Define the sampled population and target population. (b) In part (a) above, it is decided to use the results in plant A to draw conclusions about plants Band C. Define the target population. (a) What is the probability that the two observations of a random sample of two from a population with a rectangular distribution over the unit interval will not differ by more than I? (b) What is the probability that the mean of a sample of two observations from a rectangular distribution over the unit interval will be between! and.? (a) Balls are drawn with replacement from an urn containing one white and two black balls. Let X = 0 for a white ball and X = 1 for a black ball. For samples XI, X 2 , " ' , X9 of size 9, what is the joint distribution of the observations? The distribution of the sum of the observations? (b) Referring to part (a) above, find the expected values of the sample mean and sample variance. Let XI, . " Xn be a random sample from a distribution which has a finite fourth moment Define IL = @"[XI ),U 2 = var [Xd, 1L3 = @"[{XI - 1L)3], 1L4 = @"[{X1 - 1L)4].
(a)
9

X = (lIn)
(a)

2: Xi. and 8
I

= [l/{n -1)]

L (X,_X)2.
I

Does 8 2 = [l/2n(n - 1)]

L L
'''' I

{Xi - X J )2?

J'" I

266

SAMPLING AND SAMPLING DISTRIBUTIONS

vI

*(b) Find var [~)2]. *(c) Find cov [X, ~)2], and note that cov [X, ~)2] = 0 if f-t3 = o.
Possible HINT: ~(X, - f-t)2 = ~(Xi - X)2 + (1In)~~(Xj - f-t)(Xj - f-t). 6 *(a) For a random sample of size 2 from a population with a finite (2r)th moment, find tf[Mr] and var [M,], where Mr = (lIn)
(b)

L
i= 1

(Xi - XnY.

For a random sample of size n from a population with mean f-t and rth central moment f-t" show that

7 (a)

10

11

12

13

Use the Chebyshev inequality to find how many times a coin must be tossed in order that the probability will be at least .90 that X will lie between .4 and .6. (Assume that the coin is true.) (b) How could one determine the number of tosses required in part (a) more accurately, i.e., make the probability very nearly equal to .90? What is the number of tosses? If a population has u = 2 and X is the mean of samples of size 100, find limits between which X - f-t will lie with probability .90. Use both. the Chebyshev inequality and the central-limit theorem. Why do the two results differ? Suppose that Xl and X 2 are means of two samples of size n from a population with variance u 2 Determine n so that the probability will be about .01 that the two sample means will differ by more than u. (Consider Y = Xl - X 2 .) Suppose that light bulbs made by a standard process have an average life of 2000 hours with a standard deviation of 250 hours, and suppose that it is considered worthwhile to replace the process if the mean life can be increased by at least 10 percent. An engineer wishes to test a proposed new process, and he is willing to assume that the standard deviation of the distribution of lives is about the same as for the standard process. How large a sample should he examine if he wishes the probability to be about .01 that he will fail to adopt the new process if in fact it produces bulbs with a mean life of 2250 hours? A research worker wishes to estimate the mean of a popUlation using a sample large enough that the probability will be .95 that the sample mean will not differ from the population mean by more than 25 percent of the standard deviation. How large a sample should he take? A polling agency wishes to take a sample of voters in a given state large enough that the probability is only .01 that they will find the proportion favoring a certain candidate to be less than 50 percent when in fact it is 52 percent. How large a sample should be taken? A standard drug is known to be effective in about 80 percent of the cases in which it is used to treat infections. A new drug has been found effective in 85 of the first 100 cases tried. Is the superiority of the new drug well established? (If

PROBLEMS

267

the new drug were as equally effective as the old, what would be the probability of obtaining 85 or more successes in a sample of 1001) 14 Find the third moment about the mean of the sample mean for samples of size n from a Bernoulli population. Show that it approaches 0 as n becomes large (as it must if the normal approximation is to be valid). 15 (a) A bowl contains five chips numbered from 1 to 5. A sample of two drawn without replacement from this finite population is said to be random if all possible pairs of the five chips have an equal chance to be drawn. What is the expected value of the sample mean 1 What is the variance of the sample mean 1 (b) Suppose that the two chips of part (a) were drawn with replacement; what would be the variance of the sample mean 1 Why might one guess that this variance would be larger than the one obtained before? *(c) Generalize part (a) by considering N chips and samples of size n. Show that the variance of the sample mean is
u2 N n

n
where u 2 is the population variance; that is
u2 =

N-l'

~ ~
N
1=1

(i __ N_+_l) 2

16 If Xl, X 2 , X3 are independent random variables and each has a uniform distribution over (0, I), derive the distribution of (Xl X 2 )/2 and (Xl + X 2 X 3)/3. 2 17 If XI, ... , XII is a random sample from N(p., u ), find the mean and variance of

8
18

=J~(XI- X)2.
n-l

On the F distribution: (a) Derive the variance of the F distribution. [See part (d).] (b) If X has an F distribution with m and n degrees of freedom, argue that 1/ X has an F distribution with nand m degrees of freedom. (c) If X has an F distribution with m and n degrees of freedom, show that

mX/n

1 + mX/n

has a beta distribution. (d) Use the result of part (c) and the beta function to find the mean and variance of the F distribution. [Find the first two moments of mX/n = W/(l - W)].

268

SAMPLING AND SAMPLING DlSl'R.IBUTlONS

VI

19 On the I distribution: (a) Find the mean and variance of Student's t distribution. (Be careful about existence.) (b) Show that the density of a t distributed random variable approache-s the standard nonnal density as the degrees of freedom increase. (Assume that the" constant" part of the density does what it has to do.) (c) If X is t-distributed, show that X2 is F-distributed. (d) If X is I-distributed with k degrees of freedom, show that 1/(1 + X2!k) has a beta distribution. 20 Let Xl, X 2 be a random sample from N(O, O. Using the results of Sec. 4 of Chap. VI, answer the following: (a) What is the distribution of {X 2 X l )tV2? (b) What is the distribution of (Xl + X 2)2/(X2 X I )2? (c) What is the distribution of (X2 + Xl)!V (Xl - X 2)2? (d) What is the distribution of lIZ if Z = Xf! Xl? 21 Let Xl, ..... XII be a random sample from N(O, 1). Define

2: Xi k
1

and

XII - k

n-

k+l

2:

II

X,.

Using the results of Sec. 4, answer the following: (a) What is the distribution of l(Xk + X II - k)? _k ? (b) What is the distribution of kXl + (n - k)XII2 (c) What is the distribution of Xf! X~? (d) What is the distribution of XII XII? 22 Let X h , XII be a random sample from N{jJ., (12), Define

1
X n - k = n- k
n
1

k+l

2: X"

II

1 n X=- 2: X,.

S:-k = n- k and

'" (Xi 1 L.
k+l

-n) 2 X - k

..

PROBLEMS

269

Using the results of Sec. 4, answer the following: (a) What is the distribution of a-Z[(k 1)81 + (n - k - 1)8~_,,]? (b) What is the distribution of O)(X" XII-a)? (c) What is the distribution of a-z(X, - p,r~? (d) What is the distribution of 8U8~-" ? l (e) What is the distribution of (X p,)/(8lv r;)? 23 Let Zlo Z Z be a random sample of size 2 from N(O, 1) and Xl, X z a random sample of size 2 from N(1, 1). Suppose the Z,'S are independent of the X/so Use the results of Sec. 4 to answer the following: (a) What is the distribution of X Z? (b) What is the distribution of {Z. + Zz)/V[{X z - X.)i + {Zz Zlr~]/2? (c) What is the distribution of [(X. X z)Z {Z. - Zz)Z + {Z. + Zz)Z1/2? (d) What is the distribution of (Xz + XI 2)z/(Xz - XI)Z? 24 Let XI be a random variable distributed N(;, P), ; = 1, 2, 3. Assume that the random variables Xl, X z, and X 3 are independent. Using only the three random variables Xl, X z , and X3: (a) Give an example of a statistic that has a chi-square distribution with three degrees of freedom. (b) Give an example of a statistic that has an F distribution with one and two degrees of freedom. (c) Give an example of a statistic that has t distribution with two degrees of freedom. 25 Let Xl, Xz be a random sample of size 2 from the density
J(x) = le-hI(o. al)(x).

Use results on the chi-square and F distributions to give the distribution of

XI/XZ '

26 Let VI, Vz be a random sample of size 2 from a uniform distribution over the interval (O, 1). Let Y. and Y z be the corresponding order statistics. (a) For 0 < yz < 1, what is JJ'. I J'z=yz(Yllyz), the conditional density of YI given
Y z =yz? (b) What is the distribution of Y z - Y I ? 27 If XI, X z, ... XII are independently and normally distributed with the same mean but different variances af, a~., ... , a: and assuming that V = L(x,/u1)/'L.(1/uJ) and V = 'L.{X, - V)z/uf are independently distributed, show that V is normal and
'I

V has the chi-square distribution with n - 1 degrees of freedom. 28 For three samples from normal populations {with variances af. sample sizes being nl, n z , and n3, find the joint density of

a~,

and an, the

V=-

Sf Sl

and (Assume that the samples

where the 81, 8~, and Sj are the sample variances. are independent.)

270

SAMPLING AND SAMPLING DISI'RIBUTIONS

VI

29

Let a sample of size nl from a normal population (with variance have sample variance Sf, and let a second sample of size n z from a second normal population (with mean p.-z and variance a~) have mean X and sample variance S~. Find the joint density of and

an

30
31

32

33

34

(Assume that the samples are independent.) For a random sample of size 2 from a normal density with mean 0 and variance t, find the distribution of the range. (a) What is the probability that the larger of two random observations from any continuous distribution will exceed the median? (b) Genera1ize the result of part (a) to samples of size n. ConSidering random samples of size n from a population with density /(x), what is the expected value of the area under /(x) to the left of the smallest sample observation? Consider a random sample X h ., XII from the uniform distribution over the interval (p V3a, I' + V3a). Let Y 1 < ... <YII denote the corresponding order statistics. (a) Find the mean and variance of Y II - Y 1 (b) Find the mean and variance of (Y1 + Y II )/2. (c) Find the mean and variance of Yt+l if n 2k + 1, k = 0., 1, .... (d) Compare the variances of XII, Yk+h (Y1 + YII)/2. HINT: It might be easier to solve the problem for VI, .. , VII., a random sample from the uniform distribution over either (0, 1) or (-1, I), and then make an appropriate transformation. Let Xl. ... ., XII be a random sample from the density
I /(x;a,fJ) = 2P exp [-I(x- a)/PI],

where - 00 < (X < 00 and f3 > O. Compare ~he asymptotic distributions of the sample mean and the sample median. In p~rticular, compare the asymptotic variances. 35 Let XI, ... , XII be a random sample from the cumulative distribution function F(x) = {I - exp [-x/O - x)]}I(o. l){X) + 1[1. q;)(x). What is the limiting distribution of{y~lI) - all)/bll , where all log n/(1 + log n)and b;l = Oog n) (1 + logn)? What is the asymptotic distribution of Y~")? 36 Let Xl, , XII be a random sample from/(x; 8) = ()e-bl(o. q;)(x), () >0. (a) Compare the asymptotic distribution of XII with the asymptotic distribution of the sample median. (b) For your choice of {all} and {bll }, find a limiting distribution of (Y!") - all)/bll' (c) For your choice of {all} and {bll}' find a limiting distribution of (YIII> - all)/bll'

VII
PARAMETRIC POINT ESTIMATION

1 INTRODUCTION AND SUMMARY

Chapter VI commenced with some general comments about inference. There, it was indicated that a sample from the distribution of a population is useful in making inferences about the population. Two important problems in statistical inference are estimation and tests of hypotheses. One type of estimation, namely pOint estimation, is to be the subject of this chapter. The problem of estimation, as it shall be considered herein, is loosely defined as follows: Assume that some characteristic of the elements in a population can be represented by a random variable X whose density is fx( .; 0) f(' ; 0), where the form of the density is assumed known except that it contains an unknown parameter 0 (if 0 were known, the density function would be completely specified, and there would be no need to make inferences about it). Further assume that the values Xl, X2, . , Xn of a random sample Xl' X 2 , , Xn from f(; 0) can be observed. On the basis of the observed sample values X h X 2 ' , XII it is desired to estimate the value of the unknown parameter 0 or the value of some function, say r(8), of the unknown parameter. This estimation can be made in two ways. The first, called point estimation, is to let the

272

PARAMETRIC POINT ESTIMATION

vn

value of some statistic, say t(Xb ... , X n), represent, or estimate, the unknown reO); such a statistic t(Xb ... , Xn) is called a point estimator. The second, called interval estimation, is to define two statistics, say t 1(Xh .. , Xn) and tiXb , X n), where t 1 (Xh , Xn) < '2(X1 , , X n), so that (tl(XI , , XII)' t 2 (Xb .. , Xn)) constitutes an interval for which the probability can be determined that it contains the unknown reO). For example, if f( . ; 0) is the normal density, that is,
f(x; 0) = f(x; tl, u)

= 4>/l,u 2(x) =

r::L exp
y 21tu

[2! (X u It) 2] ,

where the parameter 0 is (p, u), and if it is desired to estimate the mean, that is, reO) = tl, then the statistic X r(O) = tl, and (X - 2 of reO)
= tl.

(ljn)

L Xi
I

is a possible point estimator of is a possible interval estimator

n

j8 2 /n,

X +2

j8 2 /n)

{Recall that 8 2

[lj(n - 1)]

L (Xi 1

X)2.}

Point estimation

will be discussed in this chapter and interval estimation in the next. Point estimation admits two problems: the first, to devise some meanS of obtaining a statistic to use as an estimator; the second, to select criteria and techniques to define and find a best" estimator anl0ng many possible estimators. Several methods of finding point estimators are introduced in Sec. 2. One of these, and probably the most important, is the method of maximum likelihood. In Sec. 3 several "optimum" properties that an estimator or sequence ofestimators may possess are defined. These include closeness, bias and variance, efficiency, and consistency. The loss and risk functions, essential elements in decision theory, are defined as possible tools in assessing the goodness of estimators. Section 4 is devoted to sufficiency, an important and useful concept in the study of mathematical statistics that will also be utilized in succeeding chapters. Unbiased estimation is considered in Sec. 5. The Cramer-Rao lower bound for the variance of unbiased estimators is given, as well as the Rao-Blackwell theorem concerning sufficient statistics. A brief look at invariant estimators is presented in Sec. 6. Bayes estimation is considered in Sec. 7. A Bayes estimator is given as the mean of the posterior or from the decision-theoretical viewpoint as an estimator having smallest average risk. Some results in the simultaneous estimation of several parameters are given in Sec. 8. Included is the notion of ellipsoid of concentration of a vector of point estimators and the Lehmann-Scheffe theorem. Section 9 is devoted to a brief discussion of some optimum properties of maximum-likelihood estimators.
u;

METHODS OF FINDING ESTIMATORS

273

Frequent use, of some of the distribution-theoretical results f{)r stat~stics~ which Were derived in earlier chapters~ especially Chaps. V and VI, wIll be noted throughout this chapter. After all, estimators .are.sta~isti~s, and to study properties of estimators, it is desirable to look at then dIstnbutIons.

METHODS OF FINDING ESTIMATORS

Assume that Xl' ... , Xn is a random sample from a density f(' ; e), where the form of the density is known but the parameter e is unknown. Further assume that e is a vector of real numbers, say e = (e h .. " ek ). (Often k will be unity.) We sometimes say that el , ... , ek are k parameters. We will let 9, called the parameter space, denote the set of possible values that the parameter e can assume. The object is to find statistics, functions of the observations Xl' ... , X n , to be used as estimators of the ej,j = I, ... , k. Or, more generally, our object is to find statistics to be used as estimators of certain functions, say 1'1 (e), ... , 1'r(e). of e = (elt .. , ek ). A variety of methods of finding such estimators has been proposed on more or less intuitive grounds. Several such methods will be presented. along with examples, in this section. Another method, that of the method of least squares will be discussed in Chap. X. An estimator can be defined as in Definition I.

Definition 1 Estimator Any statistic (known function of observable random variables that is itself a random variable) whose values are used to estimate 1'(e), where 1'(') is sonle function of the parameter e, is defined IIII to be an estimator of 1'(e).
An estimator is always a statistic which is both a random variable and a function. For instance, suppose Xb "', Xn is a random sample from a density f( ; e) and it is desired to estimate 1'(e), where 1'(') is some function of e. Let I(X1' ... , Xn) be an estimator of 1'(e). The estimator I( Xl, ... , Xn) can be thQught of in two related ways: first, as the random variable , say T , where T = I(X}, ... , X n), and, second, as the function 1(-, ... , .). Naturally, one needs to specify the function 1(', ... , .) before the random variable T = I(X}) ... , Xn) is defined. In all we have three types of tees : the capital Latin T, which represents the random variable I(X}) ... , X n), the small script I, which represents the function 1(', ... , .), and the small Latin t, which represents a value of T; that is, t = I(Xl' ... , x n ). Let us adopt the convention of calling the statistic (or random variable) that is used as an estimator an Uestimator" and calling a value that the statistic takes on an "estimate." Thus the word

,
VII

'274

PARAMETRIC POINT ESTIMATION

"estimator" stands for the function, and the word" estimate" stands for a value of that function; for example, Xn

=.!.. f

Xi is an estimator of a mean J.l,

ni=l

and xn is an estimate of J.l. Here Tis X n , t is Xn , and 1(' , ... , .) is the function defined by summing the arguments and then dividing by n. Notation in estimation that has widespread usage is the following: 8 is used to denote an estimate of (J, and, more generally, (81 , " ' , Ok) is a vector that estimates the v~ctor (Jl' ... , (Jk), where OJ estimates (Jj' j = 1, .,., k. If 8 is an estimate of (J, then 0 is the corresponding estimator of (J; and if the discussion requires that the function that defines both 8 and 0 be specified, then it can be denoted by a small script theta, that is, 0 = 9( Xl' ... , Xn). When we speak of estimating (J, we are speaking of estimating the fixed yet unknown value that (J has. That is, we assume that the random sample Xl, ... , Xn came from the density f(' ; (J), where (J is unknown but fixed. Our object is, after looking at the values of the random sample, to estimate the fixed unknown (J. And when we speak of estimating -r(0), we are speaking of estimating the value -r(J) that the known function -r(') assumes for the' unknown but fixed (J.

2.1

Methods of Moments

Letf(' ; (Jl' ... , (Jk) be a density of a random variable X which has k parameters (Jl' ... , (Jk' As before let It; denote the rth moment about 0; that is, J.l; = c[xr]. In general J.l; will be a known function of the k parameters (Jl' .. " (Jk' Denote this by writing J.l; = J.l;(Jl, "', (Jk)' Let Xl' .. , Xn be a random sample from the density f('; (Jl' ... , (Jk), and, as before, let Mj be the Jth sample moment; that is,
1

Mj =Form the k equations

n i= 1

LXi.
(1)

j = 1, ... , k,

in the k variables (Jl, ... , (Jk, and let 9 1, ... , 9 k be their solution (we assume that there is a unique solution). We say that the estimator (9 b ... : 9 k ), where 8j estimates (Jj' is the estimator of (Jl, .. " (Jk) obtained by the method of moments. The estimators were obtained by replacing population moments by sample moments. Some examples follow.

MElHODS of FINDING ESTIMATORS

275

EXAMPLE 1 Let Xl' .. , XII be a random sample from a normal distribution with mean Il and variance u 2 Let (Ov ( 2 ) = (p, u). Estimate the param2 eters Il and G b.y the method of moments. Recall that u = J12 - (p~)2 and Il = Ill. The method-of-moments equations become
M~ = J11

= J1~(J1, u) = J1
Jl1,(J1, u)
u

M2

J12

+ J12,

and their solution is. the following: The method-of-moments estimator of Jt is M~ X, and the method-of-moments estimator of u is' JM 2 - 2 = J(lln) xl- X2 = (Xi -X)2In. Note that the method-of-moments estimator of u given above is not jSi. IIII

JI

EXAMPLE 2 Let Xl' ... , Xn be a random sample from a Poisson distribution with parameter 1. Estimate A. There is only one parameter, hence only one equation, which is
M~

= J1~ = J1~(A) = A.

Hence the method-of-moments estimator of A is M~ = X, which says IIII estimate the population mean 1 with the sample mean X.

EXAMPLE 3 Let Xl' ... , Xn be a random sample from the negative exponential density f(x; 0) Oe- 9xI(o,oolx). Estimate 0. The method-ofmoments equation is

M~ = J1~ = J1~(0) = !; o
hence the method-of-moments estimator of 0 is I/M~

I/X.

IIII

EXAMPLE 4 Let Xl' .. , Xn be a random sample from a uniform distribution on (p. - j3u, J1 + j3u). Here the unknown parameters are two, namely II and u, which are the population mean and standard deviation. The method-of-moments equations are and

 '276'

PARAMETRIC POINT ESTIMATION

VB

hence the method-of-moments estimators are X for II and for

(J'.

We shall see later that there are better estimators of p and distribution.

(J'

for this

III!

Method-of-moments estimators are not uniquely defined. The methodof-moments equations given in Eq. (1) are obtained by using the first k raw moments. Central moments (rather than raw moments) could also be used to obtain equations whose solution would also produce estimators that would be labeled method-of-moments estimators. Also, moments other than the first k could be used to obtain estimators that would be labeled method-of-moments estimators. If, instead of estimating Uh, ... , Ok), method-of-moments estimators of, say, 'l'l(Ol' ... , Ok), ... , 'l',.(Jv . , Ok) are desired, they can be obtained in several ways. One way would be to first find method-of-moments estimates, say Ol' ... , Ok, ofO l , ... , Ok and then use riOb "', Ok) as an estimate of 7:j(Jh ... , (Jk) for j = I, "', r. Another way would be to form the equations
j

I, ... ,r

and solve them for 7: 1 , , 7:,.. Estimators obtained using either way are called method-of-moments estimators and may not be the same in both cases.

2.2 Maximum Likelihood

To introduce the method of maximum likelihood, consider a very simple estimation problem. Suppose that an urn contains a number of black and a number of white balls, and suppose that it is known that the ratio of the numbers is 3/1 but that it is not known whether the black or the white balls are more numerous. That is, the probability of drawing a black ball is either or i. If n balls are drawn with replacement from the urn, the distribution of x, the number of black balls, is given by the binomial distribution

f(x; p)

(;) p'q"

for x

0, 1, 2, ... , n,

where q = 1 - p and p is the probability of drawing a black ball. Here p = 1, or p = i. We shall draw a sample of three balls, that is, n = 3, with replacement and

METHODS' OF FINDING ESTIMATORS

277

attempt to estimate the unknown parameter p of the distribution. The estimation problem is particularly simple in this case because we have only to choose between the two numbers .25 and .75. Let us anticipate the results of the drawing of the sample. The possible outcomes and their probabilities are given below:
Outcome: x f(x; 1) f(x; -1)

0
1

1
6-4"
9

2
1.1
64

3
1.7.
64
1

117 "8.

1.1
64

9 n

'6-4"

In the present example, if we found x = 0 in a sample of 3, the estimate .25 for 1 p would be preferred over .75 because the probability it is greater than 6 4' . i.e., because a sample with x = 0 is more likely (in the sense of having larger probability) to arise from a population with p ! than from one with p i. And in general we should estimate p by .25 when x = 0 or 1 and by .75 when x = 2 or 3. The estimator may be defined as

p=

p(x) = {.25

.75

for x = 0, 1 for x = 2, 3.

The estimator thus selects for every possible x the value of p, say /1, such that
f(x;

/1) > f(x; p'),

where p' is the alternative value of p. More generally, if several alternative values of p were possible, we might reasonably proceed in the same manner. Thus if we found x = 6 in a sample of 25 from a binomial population, we should substitute all possible values of p in the expression for 0 <p< I
(2)

anti choose as our estimate that value of p which maximizedf(6; p). For the given possible values of p we should find our estimate to be 265- The position of its maximum value can be found by putting the derivative of the function defined in Eq. (2) with respect to p equal to 0 and solving the resulting equation for p. Thus,

~f(6") dp ,p

= (25) 6

P5 (l - p) 18 [6(1 - p)

19p],

278

PA~C POINT ESTIMATIoN

vn

and on putting this equal to 0 and solving for p, we find that p 0, I, 265 are the roots. The first two roots give a minimum, and so our estimate is therefore p 265' This estimate has the property that
f(6;

P) > f(6;

p'),

where pi is any other value of p in the interval 0 p ~ 1. In order to define maximum-likelihood estimators, we shall first define the likelihood function. Definition 2 Likelihood function The likelihood function of n random variables XI' X 2 , " ' , Xn is defined to be the joint density of the n random variables, say fXI, ... ,x.lxl , , Xn; 0), which is considered to be a function of 8. In particular, if XI, ... , Xn is a random sample from the density f(x; 0), then the likelihood functionisf(xl; 8)f(X2; 8) '" 'f(xn; 8). IIII Notation To remind ourselves to think of the likelihood function as a function of 8, we shall use the notation L(O; Xv , xn) or L(. ; Xh .. , xn) for the likelihood function. IIII The likelihood function L(O; XI' , xn) gives the likelihood that the random variables assume a particular value Xl, X2, ... , Xn . The likelihood is the value of a density function; so for discrete random variables it is a probability. Suppose for a moment that 8 is known; denote the value by 00 , The particular value of the random variables which is "most likely to occur" is that value x~, Xl' ... , x~ such that fXI, ... , xn(Xf, ... , Xn; ( 0) is a maximum. For example, for simplicity let us assume that n = 1 and Xl has the normal density with mean 6 and variance 1. Then the value of the random variable which is most likely to occur is Xl = 6. By" most likely to occur" we mean the value x~ of Xl such that cP6,I(xD>cP6,I(XI)' Now let us suppose that the joint density of n random variables is fx!, ... xn(xI , , Xn; 8), where 8 is unknown. Let the particular values which are observed be represented by Xl, X2' ., ., x~. We want to know from which density is this particular set of values most likely to have come. We want to know from which density (what value of 0) is the likelihood largest that the set x~, ... , x~ was obtained. In other words, we want to find the value of 0 in 9, denoted by {J, which maximizes the likelih~~d function L(O; x~, ... , x~. The value {J which maximizes the likelihood function is, in general, a function of Xf, , Xn, say (J = 9(Xh X2,"'" xJ. When this is the case, the random variable = 8(Xf, X 2 , , Xn) is called the maximumlikelihood estimator of O. (We are assuming throughout that the maximum of the likelihood function exists.) We shall now formalize the definition of a maximum-likelihood estimator.

METHODS OF FINDING ESTIMATORS

279

Definition 3 Maximum-likelihood estimator Let L(O) =L(O; Xl' ... , Xli) be the likelihood function for the random variables Xl' X 2 , "', Xli' If lJ [where (J = 9(Xl' X2, ... , Xli) is a function of the ~bservations Xl, ... , Xli] is the value of 0 in e which maximizes L(O), then e = 8(Xl' X 2 , , Xli) is the maximum-likelihood estimator of O. {} = 8(xI' ... , Xli) is the maximum-likelihood estimate of 0 for the sample Xl' .. " Xli' 1//1

The most important cases which we shall consider are those in which Xb X 2 , .,., Xli is a random sample from some density f(x; 0), so that the likelihood function is
L(O)

= f(Xl ; O)f(X2; 0) ... f(x

lI ;

0).

Many likelihood functions satisfy regularity conditions; so the maximumlikelihood estimator is the solution of the equation

dL(O) dO

= O.

Also L(O) and log L(O) have their maxima at the same value of 0, and it is sometimes easier to find the maximum of the logarithm of the likelihood. If the likelihood function contains k parameters, that is, if
L(Ol' O2 , . .. , Ok)

= nf(Xj; Oh O2 ,
i= 1

II

Ok),

then the maximum-likelihood estimators of the parameters 01 , O 2 , ... , Ok are the random variables 0 1 = 9t (Xb " ' , Xn), O2 = 82 (Xl , ... , XJ, ... , 0 k = 8lXb ... , Xli)' where {}l' (}2, "', 6k are the values in e which maximize L(Ob Oz, ... , Ok)' If certain regularity conditions are satisfied, the.point where the likelihood is a maximum is a solution of the k equations

aLeOb ... , Ok)

aO
=
k

O.

In this case it ~ay also be easier to work with the logarithm of the likelihood. We shall Illustrate these definitions with some examples.

280.

pAItAMETlUC POINT ESTIMATION

VII

EXAMPLE 5 Suppose that a random sample of size n is drawn from the Bernoulli distribution

o ~ p .s; 1 and q =
The sample values Xh likelihood function is
X2, " ' , XII

1 - p.

will be a sequence of Os and Is, and the

L(p) =

i= 1

n
II

ylql-XI = pf.Xiqn-f.xi,

and if we let

we obtain *log L(P) = y log p + (n - y) log q and

dlogL(p) y n- y =- , dp p q

remembering that q = 1 - p. On putting this last expression equal to 0 and solving for p, we find the estimate

P=

y
n

- = -

2. Xi = n

x,

(3)

which is intuitively what the estimate for this parameter should be. It is also a method-of-moments estimate. For n = 3, let us sketch the likelihood function. Note that the likelihood function depends on the x/s only through 2. Xi; thus the likelihood function can be represented by the following four curves:

2. Xi = 0) = (1 - p)3 Ll L(P; 2. Xi = 1) = P(1 - p)2 L2 = L(p; 2. Xi = 2) = p2(1 - p) L3 = L(P; 2. Xl = 3) = p3,

Lo = L(P;

which are sketched in Fig. 1. Note that the point where the maximum of each of the curves takes place for 0 p 1 is the same as that given in 11I1 Eq. (3) when n = 3.
,.. Recall that log x means loge x.

METHODS OF FINDING ESTIMATORS

281

L(p)
1

FIGURE 1

EXAMPLE 6 A random sample of size n from the normal distribution has the density

i=l

Ii J2nu 1 e-(1/2a

)(xi-Jt)2

= (~)n/2 exp [- ~ I (Xi 2nu 2u

tl)2] .

The-logarithm of the likelihood function is

I (x - tl) 2 L lie = - n -log 2n - n -log u 2 - - I 2 2 2 2u i ,

where u > 0 and - 00 < tl < 00. To find the location of its maximum, we compute

and
aLate au 2

= - 2. u 2 + 2u4 I

n 1

(Xi -

tl) ,

and on putting these derivatives equal to 0 and solving the resulting equations for tl and u 2 , we find the estimates
(4)
"'2 U

1" =L. (Xi n

- 2, - X)

(5)

which turn out to be the sample moments corresponding to tl and u 2

I111

282

PAR.AMETRIC pOINT ESTIMATION

VII

EXAMPLE 7 Let the random variable X have a uniform density given by

f(x; 0) = I[o-t,o+-u(x),

where - 00 < 0 < 00; that is~ for a sam pIe of size n is
L(O;
X h ,

e = real
n
i ;

line.
II

The likelihood function

x,,)

= nf(x
1=1

0) =

i=l

nI

lo -

t , O+t1(Xj)

= I h 'n-t,)ll+tl.),

(6)

where Yl is the smallest of the observations and Y1I is the largest. The last equality in Eq. (6) follows since
i= 1

." n IlO-t,o+u(xi) is unity if and on1y

if all Xl' , X" are in the interval [0 - t, 0 + t], which is true if and only if 0 - t < YI and Y" s 0 + t, which is true if and only if y" - t 0 <YI + t. We see that the likelihood function is either 1 (for Yn - t < 0 < YI + i) or 0 (otherwise); hence any statistic with value iJ satisfying Y1I - i < {j YI + t is a maximum-likelihood estimate. Examples are Yn - i, Yl + i, and llil + Y1I)' This latter is the midpoint between Yn -!- and Yt + i, or the midpoint between Yl and Y1I' the smallest and largest observations. / / //

EXAMPLE 8 Let the random variable X have a uniform distribution with density given by
1

f(x; 0) = f(x; Jl, 0') = 2J3a hll-./3o-,Il+,/3o-1(x),

where - 00 < p < 00 and 0' > O. (Recall Example 4.) hood function for a sample of size n is
L(p, 0';
Xl"'" Xn)

Here the likeli-

(~ )"fIJ11l - 43 o- 1l+.J3o-lx ,) 30'

i=l

(J

3 0') \ . -'/3 ., ,.J(Y ,)I[j>., +,/ J<y,)

'1[(.-,.)/,/3,00)(0')1[<1.-.)/,/3, oo)(O')/[j>" OO)(Y,).

(d3.,.)

where YI is the smallest of the observations and Yn is the largest. The likelihood function is (2J3a)-n in the shaded area of Fig. 2 and 0 elsewhere. (2J3a) -n within the shaded area is clearly a maximum when 0'

METHODS OF FINDING ESTIMATORS

283

FIGURE 2

is smallest, which is at the intersection of the lines J.! - )3 (J = Yl and J1 + (J = Yn. Hence the maximum-likelihood estimates of J.! and (J are

J3

.a = '2 (y 1 + Y n)
and
lJ
1 J3 (yn 2 3

(7)

Yl),

(8)

which are quite different from the method-of-moments estimates given in Example 4. IIII The above four examples are sufficient to illustrate the application of the method of maximum likelihood. The last two show that one must not always rely on the differentiation process to locate the maximum. The function L(O) may, for example, be represented by the curve in Fig. 3, where the actual maximum is at fJ, but the derivative set equal to 0 would locate 0' as the maximum. One must also remember that the equation 8LI80 = 0 locates minima as well as maxima, and hence one must avoid using a root of the equation which actually locates a minimum. We shall see in later sections (especially Sec. 9 of this chapter) that the maximum-likelihood estimator has some desirable optimum properties other than the intuitively appealing property that it maximizes the likelihood function. In addition, the maximum-likelihood estimators possess a property which is sometimes called the invariance property of maximum-likelihood estimators. A little reflection on the meaning of a single-valued inverse will convince one of the validity of the following theorem.

284

PARAMETRIC POINT ESTIMATION

VII

L(O)

FIGURE 3

~--~--------~--------------~O

0'

Theorem 1 Invariance property of maximum-likelihood estimators Let = 8(Xl' X 2 , , Xn) be the maximum-likelihood estimator of 0 in the density I(x; 0), where 0 is assumed unidimensional. If r( .) is a function

with a single-valued inverse, then the maximum-likelihood estimator of r(O) is r(e). 1111 For example, in the normal density with 110 known the maximum-likelihood estimator of (J2 is

By the invariance property of maximum-likelihood estimators, the maximumlikelihood estimator of (J is

Similarly, the maximum-likelihood estimator of, say, log

(J2

is

log [~.tl (Xi -

1'0)2].

The invariance property of maximum-likelihood estimators that is exhibited in Theorem 1 above can and should be extended. Following Zehna [43], we extend in two directions: First. 0 will be taken as k-dimensional rather than unidimensional, and, second, the assumption that r() has a single-valued inverse will be removed. It can be noted that such extension is necessary by considering two simple examples. As a first example, suppose an estimate of the variance, namely O(l - 0), of a Bernoulli distribution is desired. Example 5 gives the maximum-likelihood estimate of 0 to be X, but since O(l - 0) is not a one-to;'one function of 0, Theorem 1 does not give the maximum-likelihood

MEmoDS OF f'INDING ESTIMATORS

285

estimator of 8(1 - 8). Theorem 2 below will give such an estimate, and it will be x(1 - x). As a second example, consider sampling from a normal distribution whete both It and a2 are unknown, and suppose an estimate of C[X2] = It'}. + (12 is desired. Example 6 gives the maximumMlikelihood estimates of J1 and a2 , but It'}. + (12 is not a one-to-one function of J.i and 0'2, and so the maximum-likelihood estimate of It'}. + 0'2 is not known. Such an estimate will be obtainable from Theorem 2 below. It will be x2 + (lIn) L (Xl' - X)2. Let 0 = (01 , , OJ be a k-dimensional parameter, and, as before, let e denote the parameter space. Suppose that the maximum-likelihood estimate of t(O) = (tl (0), ... , 1'r(O, where 1 < r < k, is desired. Let T denote the range T is an rMdimensional space of the transformation t() = (tl (.), ... , t r ( sup L(O; Xl' . , Xn) M( ; Xh . , Xn) space. Define M( t; Xl' ... , Xn) =

{6:.(6)=.}

is called the likelihood function induced by t( ).* When estimating 0 we maximized the likelihood function L(O; X h . , xn) as a function of 0 for fixed Xb " ' , x,,; when estimating t = 1'(0) we will maximize the likelihood function induced by t(), namely M(1'; Xl' . , X n), as a function of t for fixed Xh , X n Thus, the maximum-likelihood estimate of t = t(O), denoted by t, is any value that maximizes the induced likelihood function for fixed Xl' .. , x"; that is, t is such that M(t; Xl' .. , xn) M(t; Xl' .. , X,,) for all l' E T. The invariance property of maximum-likelihood estimation is given in the following theorem. Theorem 2 Let 8 = (81 , ... , 8 k ), where OJ = 8/Xl , , Xn)' be a maximum-likelihood estimator of 0 = (01 , , Ok) in the density f( ; 01, ... , Ok)' If t(O) = (1'1(0), ... , tr(O for 1 r k is a transformation of the parameter space ~, then a maximum-likelihood estimator of t(O) = (1'1(0), ... , tr(O is 1'(0) = (t l (8), ... , t,.(0. [Note that tiO) = ti~l' ... , O}.); so the maximum-likelihood estimator of tj(Ol' ... , Ok) is t i 0 1 , , 0 k), j = 1, .. ., r.] be a maximum-likelihood estimate of 0= (Oh ... , Ok) It suffices to show that M(1'(fJ); Xl' .. , Xn) > M( t; Xl' . , xn) for any l' E T, which follows immediately from the inequality M(t; Xl''' Xn) = sup L(O; Xl"'" X) sup L(O X . X) {8: T(8)=t} " / f e e ' 1, 'n = L(fJ; x[, ... , Xn) = sup L(O; x[, ... , Xn) = M(1'(tJ) X X). IIII (8; T(8)=T(tJ)} '1, ,n
PROOF
1 , ,

Let

e= (e

eJ

*The notation 't IS ' usua II y use d In . . "sup" is used here. and elsewhere I'n th' IS b 00 k ,as 1 mathematIcs. For those readers Who are not acq a' t d 'th th' t t' h . I t 'f" .. I u In e WI IS no a lon, not mue IS OS I sup IS rep aced by "max" h ' . . . . were max IS an abbreVIatIon for maxImum.

286

P.AIlANETRIC POINT ESTIMATION

VII

It is precisely this property of invariance enjoyed by maximum-likelihood

estimators that allowed us in our discussion of maximum-likelihood estimation to consider estimating (Ott ... , Ok) rather than the more general T 1 (OI' ... , Ok), ... , 1:l'(Oh .. " Ok)'

EXAMPLE 9 In the normal density, let = (Oh ( 2 ) = (j1., (i2). Suppose T(O) = J1 + Zq u, where Zq is given by tfJ(Zq) = q. T(O) is the qth quantile. According to Theorem 2, the maximum-likelihood estimator of T(O) is
X

+ zqj(l/n) I (X~ -

X)2.

1I11

2.3

Other Methods

There are several other methods of obtaining point estimators of parameters. Among these are (i) the method of least squares, to be discussed in Chap. X, (ii) the Bayes method, to be discussed later in. this chapter, (iii) the minimum-chi-square method, and (iv) the minimum-distance method. In this subsection we will briefly consider the last two. Neither will be used again in this beok. Minimum-chi-square method Let Xl' ... , Xn be a random sample from a density given by I x(X; 6), and let fJ\, "', fI' k be a partition of the range of X. The probability that an observation falls in cell fI' j ' j = I, "', k, denoted by Pj(O), can be found. For instance, if Ix(x; 0) is the density function of a continuous random variable, then Pj(O) = P[X falls in cell fl'j] = SYJ Ix(x; 6) dx. Note that

p/6)

= 1. Let the random variable N j denote the number of

j=1

X/s in the sample which falls in cell fl'j' j

= I,

... , k; then

Nj

= n,

the

j=1

sample size.

Form the following summation:

where nj is a value of N j The numerator ofthejth term in the sum is the square of the difference between the observed and the expected number of observations falling in cell fj' j The minimum-chi-square estimate of 6 is that tJ which minimizes x2 It is that 0 among all possible O's which makes the expected number of observation in cell fl'j "nearest" the observed number. The minimumchi-square estimator depends on the partition fl'1' , fj' k selected.

ME1HODS Oil FINDING ESTIMATORS

287

EXAMPLE 10 Let Xl' .. ., XII be a random sample from a Bernoulli distribution; that is, fx(x; 0) = OX(l - O)l-X for x 0, 1. Take N, = the Here the range of the f bservations equal to j for j = 0, 1. . numberoo . f h b observation X is partitioned into the two sets conslstmg 0 t e num ers
and 1 respectively.

x - " jf:o
2

[n.
J

nptO )] 2 [no - n(l - 0)]2 = nptO) n(l - 0)

-

(n I

nO)2 n0
1
0(1 - 0)

[n - n 1
-

n(l - 0)]2 n(l - 0)

+~I

nO)2 nO

= (n 1 n

nO)2

The minimum of X2 as a function of (} can be found by inspection by noting that x2 = 0 for 0 = n1/n. Hence (j = ntln. For this example there was only one choice for the partition [/.' ., ., [/ k' The estimator found is the same as what would be obtained by either the method of moments or maximum likelihood. IIII Often it is difficult to locate that () which minimizes X2; hence, the denominator np/O) is sometimes changed to nj (if nj = 0, unity is used) forming a

modified X2

([n) - np/O)]2Inj}. The modified minimum-chi-square estimate

j= I

of 0 is then that () which minimizes the nlodified X2 Minimum-distance method Let XI' ... , Xn be a random sample from the distribution given by the cumulative distribution function Fx(x~ 0) = F(x~ 0), and let d(F, G) be a distance function that measures how "far apart" two cumulative distribution functions F and G are. An example of a distance function is d(F, G) = sup IF(x) -"G(x) I, which is the largest vertical distance between F
x

and G. See Fig. 4. The minimum-distance estimate of 8 is that (j among all possible for which d(F(x; 8), F ,ix is mil~Jmized, where F n(x) is the sample cumulative distribution function. Thus, 8 is chosen so that F(x; (j) will be "closest" to F,,(x) , which is desirable since we saw in Subsec. 5.4 of Chap. VI that for a fixed argument x the sample cumulative distribution function has the same distribution as the mean of a binomial distribution; hence, by the law of large numbers F "(x) "converges" to F(x). The minimum-distance estimator might be intuitively appealing, but it is almost always difficult to find since locating {} which minimizes d(F(x; 8), F,,(x) is seldom easy. The following example is an exception.

288

PARAMETRIC POINT ESTIMATION

VII

1
d(F. G)

~--~--------------------------------~X

FIGURE 4

EXAMPLE 11 Again let Xl' ... , Xn be a random sample from a Bernoulli distribution; then
F(x; B)

= (I -

B)/[o, 1)(x)

+ 1[1, OCllx),-

where 0 < B :s:: 1. Then

Let nj = the number of observations equal to};} = 0, 1.

no F ,.(x) = -;;: 1[0, l/X) +
11,

OCl)(X).

Now if the distance function d(F, G) = sup IF(x) - G(x) I is used, then
x

d(F(x; 8), Fix is minimized if 1 - 0 is taken equal to noln or 0 = ndn = xi/no Hence &= x. IIII

For a more thorough discussion of the minimum-chi-square method, see Cramer [11] or Rao [17]. The minimum-distance method is discussed in Wolfowitz [42].

3 PROPERTIES OF POINT ESTIMATORS

We presented several methods of obtaining point estimators in the preceding section. All the methods were arrived at on a more or less intuitive basis. The question that now arises is: Are some of many possible estimators better, in some sense, than others? In this section we will define certain properties. which an estimator mayor may not possess, that will help us in deciding whether one estimator is better than another.

3.1

Closeness

If we have a random sample Xv ... , Xn from a density, say f(x; 8), which is known except for 0, then a point estimator of 1'(8) is a statistic, say t(XI' ... , XII)' whose value is used as an estimate of 1'(8). We will assume here that 1'(0) is a

PRoPERnES OF POINt ESTIMATORS

289

real-valued (not a vector) function of the unknown parameter 8. [Often 1'(8) will be 6 itself.) 'ldeallY7 we.. would like the value of !(XI , . , Xn) to be the unknown 1'(0), but this is not possible except in trivial cases, one of which follows.

EXAMPLE 12 Assume that one can sample from a density given by

f(x; e)

= I(8--hIH-!)(X),

where is known to be an integer. That is, S, the parameter space, consists of all integers. Consider estimating 8 on the basis of a single observation Xl' If !(x I ) is assigned as its value the integer nearest Xl' then the statistic or estimator t(XI ) will always correctly estimate 8, In a sense, the problem posed in this example is really not statistical since one knows the value of () after taking one observation. //1/ Not being able to achieve the ultimate of always correctly estimating the unknown 't'(), we _look for an estimator t( Xl' ... , X,,) that is "close" to 1'(e). There are several ways of defining" close," T = !(XI' ... , Xn) is a statistic and hence has a distribution, or rather a family of distributions, depending on what () is. The distribution of Ttells how the values t of Tare distributed, and we would like to have the values of T distributed near 1'(0); that is, we would like to select t(, ,.,' .) so that the values of T = t(XI' ... , Xn) are concentrated near 't'()). We saw that the mean and variance of a distribution were, respectively, measures of location and spread. So what we might require of an estimator is that it have its mean near or equal to 1'(0) and have small variance. These two notions are explored in Subsec. 3.2 below and then again in Sec. 5. Rather than resorting to characteristics of a distribution, such as its mean and variance, one can define what" concentration" might mean in terms of the distribution itself. Two such definitions follow. More concentrated and most concentrated Let T = t(Xh ... , Xn) and T' = t'(XI' ... , Xn) be two estimators of 1"(8). T' is called a more concentrated estimator of 1"(8) than T if and only if P8[1"(e) -). < T' < 1"(0) +).] > P8 [1'(O) -). < T < 1"(8) +).] for all ). > 0 and for each 0 in S. An estimator T* = t*(XI , " ' , X") is called most /11/ concentrated if it is more concentrated than any other estimator, Definition 4

Remark The subscript 8 on the probability symbol P e['] is there to emphasize that, in general, such probability depends on e. For instance,

290

pAIlAMETRIC poINT ESTIMATION

vn

Pe[1"(O) - ..l < T < 1"(0) + ..l], the event {r(O) - ..l < T:5: 1"(0) +..l} is described in terms of the random variable T, and, in general, the distriIII

bution of T is indexed by O.

IIII

We see from the definition that the property of most concentrated is highly desirable (Pitman [41], in defense of his calling a 'most concentrated estimator best, stated that such an estimator is undeniably best"); unfortunately, most concentrated estimators seldom exist. There are just too many possible estimators for anyone of them to be most concentrated. What is then sometimes done is to restrict the totality of possible estimators under consideration by requiring that each estimator possess some other desirable property and to look for a best or most concentrated estimator in this restricted class. We will not pursue the problem of finding most concentrated estimators, even within some restricted class, in this book. Another criterion for comparing estimators is the following one.
H

Definition 5

Pitman-closer and Pitman-closest Let T = I(XI' ... , X,,) and T' = 1'( Xl' ... , X n) be two estimators of 1'(8). T' is called a Pitmancloser estimator of 1"(8) than T if and only if for each 8 in

e.

An estimator T* is called Pitman-closest if it is Pitman-closer than any other estimator. IIII The property of Pitman-closest is, like the property of most concentrated, desirable, yet rarely will there exist a Pitman-closest estimator. Both Pitman-closer and more concentrated ,are intuitively attractive properties to be used to compare estimators, yet they are not always useful. Given two estimators T and T', one does not have to be more concentrated or Pitman-closer than the other. What often happens is that one, say T, is Pitman-closer or more concentrated for some 0 in 8, and the other T' is Pitman-closer or more concentrated for other 0 in e; and since 8 is unknown, we cannot say which estimator is preferred. Since Pitman-closest estimators rarely exist for applied problems, we will not devote further study to the notion in this book; instead, we will consider other ways of measuring the closeness of an estimator to 1"(0). Competing estimators can be compared by defining a measure of the closeness of an estimate to the unknown 1"(8). An estimator T' = I'(XI' ... , X,,) of 1"(8) will be judged better than an estimator T = t( Xl' ... , X,,) if the measure of the closeness of T' to 1"(0) indicates that T' is closer to 1"(0) than T. Such concepts of closeness will be discussed in Subsecs. 3.2 and 3.4.

PROPERTIES OF POINT ESTIMATORS

291

In the above we were assuming that n, the sample size, was fixed. Still another meaning can be affixed to" closeness" if one thinks in terms of increasing sample size. It seems that a good estimator should do better when it is based on a large sample than when it is based on a small sample. Consistency and asymptotic efficiency are two properties that are defined in terms of increasing sample size; they are considered in Subsec. 3.3. Properties of point estimators that are defined for a fixed sample size are sometimes referred to as small-sample properties, whereas properties that are defined for increasing sample size are sometimes referred to as large-sample properties.

3.2 Mean-squared Error

A useful, though perhaps crude, measure of goodness or closeness of an estimator t(XI' ... , Xn) of 'l{O) is what is called the mean-squared error of the

estimator. Definition 6 Mean-squared error Let T = t(Xl , " ' , Xn) be an estimator of 1'(8). Go[[T - 1'(8)]2] is defined to be the mean-squared error of the estimator T = t(XI' ... , Xn) IIII

Notation Let MSE,(O) denote the mean-squared error of the estimator

T
=

t(XI ,

. ,

Xn) of 1'(8).

IIII

Remark The subscript 8 on the expectation symbol G(J indicates from which density in the family under consideration the sample came. That is,
G6 [[T - 1'(8)]2]

= Ge[[t(Xh "', Xn) =

1'(O)j2]

J... J[t(XI' .. , xn) -

1'(O)]~(XI; 0) f(x,,; 8) dxl

dx n ,

where f(x; 8) is the probability density function from which the random IIII sample was selected. The name" mean-squared error" can be justified if one first thinks of the difference t - 1'(0), where t is a value of Tused to estimate 1'(8), as the error made in estimating 1'(8), and then interprets the U mean" in "mean-squared error" as expected or average. To support the contention that the meansquared error of an estimator is a measure of goodness, one merely notes that ife[[T - 1'(O)Fl is a measure of the spread of T values about 1'(0), just as the' variance of a random variable is a measure of its spread about its mean. If we

VII

- -.,.--- ...........
MSE, (6) MSE'l (6)

FIGURE 5

________-1----_. 8

were to compare estimators by looking at their respective mean-squared errors, naturally we would prefer one with small or smallest mean-squared error. We could define as best that estimator with smallest mean-squared error, but such estimators rarely exist. In general, the mean-squared error of an estimator depends on O. For any two estimators Tl = 11(X1, ... , Xn) and T2 = 1 2(Xb ... , Xn) of r(O), their respective mean-squared errors MSEt}(O) and MSEtiO) as functions of 0 are likely to cross; so for some 0, 11 has smaller MSE, and for others 12 has smaller MSE. We would then have no basis for preferring one of the estimators over the other. See Fig. 5. The following example shows that except in very rare cases an estimator with smallest mean-squared error will not exist.

EXAMPLE 13 Let Xl' ... , Xn be a random sample from the density f(x; 0), where 0 is a real number, and consider estimating 0 itself; that is, r(O) = O. We seek an estimator, say T* = 1*(Xb ... , X n), such that MSEt*(O) < MSEA.O) for every 0 and for any other estimator T = I(Xb ... , Xn) of O. Consider the family of estimators T90 = 190(X1, ... , Xn) 00 indexed by 00 for 00 E e. For each 00 belonging to 8, the estimator 160 ignores the observations and estimates 0 to be 00 , Note that

MSEtl1 /O)

6"9[[1 90(X1, ... , Xn) - 0]2]

-

= $9[(0 0

0)2]

= (0 0

0)2;

so MSEtO()(Oo) = 0; that is, the mean-squared error of 180 evaluated at 0= 00 is O. Hence, if there is to exist an estimator T* = 1*(X1, ... , Xn) satisfying MSEt*(O) < MSEt(O) for every 0 and for any estimator I, MSEt*(O) - O. [For any 0 0 , MSEt*(Oo) = 0 since MSEt*(Oo) < MSE t 00 (0 0 ) = 0.] In order for an estimator 1* to -have its meansquared error identically 0, it must always estimate 0 correctly, which means that from the sample you must be able to identify the true parameter IIII value.

PROPERTIES OF POINT ESTIMATORS

293

One reason for being unable to find an estimator with uniformly smallest mean-squared error is that the class of all possible estimators is too largeit includes some estimators that are extremely prejudiced in favor of particular O. For instance, in the example above t(Jo(X1 , , XII) is highly partial to Bo since it always estimates 0 to be 00 One could restrict the totality of estimators by considering only estimators that satisfy some other property. One such property is that of unhiasedness. Definition 7 Unbiased An estimator T = t(Xl' ... , XII) is defined to be an unbiased estimator of 'r(B) if and only if for all 0
E

e.

/11/

An estimator is unbiased if the mean of its distribution equals -,;(B) , the function of the parameter being estimated. Consider again the estimator t(Jo(X" "', XII) = eo of the above example; tf(J[t60(X1 , ... , XII)] = tf(J[Bo] = 60 :# 6; so t(Jo(X1 , , XII) is not an unbiased estimator of B. Ifwe restricted the totality of estimators under consideration by considering only unbiased estimators, we 'Could hope to find an estimator with uniformly smallest mean-squared error within the restricted class, that is, within the class of unbiased estimators. The problem of finding an unbiased estimator with uniformly smallest meansquared error among all unbiased estimators is dealt with in Sec. 5 below. Remark MSEt (6) = var [T] + {'r(B) - tfe[T]}2. So if Tis an unbiased estimator of 'r(B), then MSEAB) = var [T].
~

(9)

PROOF

MSEAO)

= 8 e[[T - -,;(B)]2] = tfe[T - 8 e[T])- {'r(e) - tfe[T]})2]

= 8 e[(T - 8 e[TD2] - 2{'r(O) - 8 e[T]}cf(J[T - cfe[T]] + tfe[{-,;(B) - 8 e[T]}2] = var [T] + {-,;(B) - 8(J{T]}2.

IIII

The term -,;(B) - tfe[T] is called the bias of the estimator T and can be either positive, negative, or zero. The remark shows that the mean-squared error is the sum of two nonnegative quantities; it also shows how the meansquared error, variance, and bias of an estimator are related.

EXAMPLE 14 Let Xl~ ... , XII be a random sample from/(x; B) = p.,al(x). Recall that the maximum~likelihood estimators of p and (12 are, respectively, X and (lin) L (Xi - X)2. (See Example 6.) Now 8(J[X] = p; so

294

PARAMETRIC POINT ESTIMATION

VII

X is an unbiased estimator of 11, and hence the mean-squared error of X = ct8 [(X - 11)2] = var [X] = a 2ln. We know that ct 9 [8 2] = a 2 ; so

ct 9 [(1/n)

L (Xi -

X)2]

= =

[(n - 1)ln]ct9 [[1/(n - 1)]

L (Xi -

X)2]

[(n - 1)ln]ct9 [8 2 ] = [(n - 1)ln]a2

Hence the maximum-likelihood estimator of a 2 is not unbiased. mean-squared error of (lin) L (Xi - X)2 is ct [[(lln) L (Xi - X)2 - a2]2]
9

The

= var
=

[(lIn) L (Xi
2 ]

- X)2]

+ {a 2 n

ct 9 [(1ln)

L (Xi -

X)2]}2

(n - 1)2 var [8
n2

-1 )2 + (n a2 _ - - a2
4) + a -,
4

(n - 1)2 1( n- 3 = - 114 a 2 n n n- 1

n2

using Eq. (10) of Theorem 2 in Chap. VI.

IIII

Remark For the most part, in the remainder of this book we will take the mean-squared error of an estimator as our standard in assessing the IIII goodness of an estimator. 3.3 Consistency and BAN

In the previous subsection we defined the mean-squared error of an estimator and the property of unbiasedness. Both concepts were defined for a fixed sample size. In this subsection we will define two concepts that are defined for increasing sample size. In our notation for an estimator of ,(8), let us use Tn = liX1, ... , Xn), where the subscript n of I indicates sample size. Actually we will be considering a sequence of estimators, say Tl = 11(X1), T2 = 1 2(X1, X 2) T3 = 1 3(X1, X 2 , X 3), ... , Tn = I iX1, ... , X n), .... An obvious example is
Tn

= I iX1,

... , Xn)

= X n = (lin) L Xi'
i= 1

Ordinarily the functions In in the

sequence will be the same kind of function for each n. When conSidering a sequence of estimators, it seems that a good sequence of estimators should be one for which the values of the estimators tend to get closer to the quantity being estimated as the sample size increases. The following definitions formalize this intuitively desirable notion of limiting closeness.

Definition 8 Mean..squared-error consistency Let T1, T2 , ... , Tn ... be a sequence of estimators of ,(8), where Tn = liXl' ... , Xn) is based on a sample of size n. This sequence of estimators is defined to be a

PROPERTIES OF pOINT ESTIMATORS

295

mean-squared-error consistent sequence of estimators of 1:(8), if and only if lim 8 e[[T,. - ,;(8)]2] = 0 for all 8 in 9. IIII

Remark

Mean..squared-error consistency implies that both the bias and the variance of Tn approach 0 since 8 9 [[Tn - ,(8)]2] = var [Tn] + {,(O) - 8 e[Tn]}2. IIII In sampling from any density having mean J1 and variance

EXAMPLE 15

(12~ let Xn = (lIn) L Xi be a sequence of estimators of J.l and

i= 1

,.

S; =

[l/(n - 1)] L (Xi - X n) be a sequence of estimators of (12.

i= 1

8 [(X n - J.l)2] =

var [X n] = (121n -40 as n -4 00 ; hence the sequence {X n} is a mean-squarederror consistent sequence of estimators of J1. 0"[(8; 0-

2)2]

) .... 0 = var [8;] = ~ (P4 -: ~ 0-4

as n -+ 00, using Eq. (10) of Chap. VI; hence the sequence {S;} is a

mean-squared-error consistent sequence of estimators of (12. Note that if Tn = (lIn) I (Xi - X)2, then the sequence {Tn} is also a mean-squaredIIII error consistent sequence of estimators of (12. There is another weaker notion of consistency given in the following definition. Definition 9 Simple consistency Let T1 , T 2 , , Tn, ... be a sequence of estimators of 1:"(0), where Tn = t n(Xb ... , Xn). The sequence {Tn} is defined to be a simple (or weakly) consistent sequence of estimators of ,(8) if for every 8 > 0 the following is satisfied:

,.-+

lil!l Pe[,;(8) 00

< Tn < 1:"(8) +

8]

=1

for every 0 in El.

I1II

Remark If an estimator is a mean-squared-error consistent estimator,

it is also a simple consistent estimator, but not necessarily vice versa.
PROOF

Pe[1:"(O) -

< Tn < 1:(8) + e] = P[I Tn - ,(0) I <

8]

= Pe[[Tn - 1:"(8)]2 <

8 2]

> I _ tfe[[Tn ~ 1:(0)j2]

e

296

PARAMETRIC POINT ESTIMATION

VII

by the Chebyshev inequality. As n approaches infinity, $8[[Tn - 1"(0)]2] approaches 0; hence lim Po[1"(e) - e < Tn < 1"(0) + e] = 1. fill
n-oo

We close this subsection with one further large-sample definition.

Definition 10 Best asymptotically normal estimators (BAN estimators) A sequence of estimators Ti, ... , T:, ... of 1"(8) is defined to be best asymptotically normal (BAN) if and only if the following four conditions
are satisfied: (i) The distribution of 1"(8)] approaches the normal distribution with mean 0 and variance U*2(0) as n approaches infinity. (ii) For every e > 0, lim po[1
n-oo

In[T: =

T: - 1"(0)1 > e]

for each 0 in

e.

(iii)

Let {Tn} be any other sequence of simple consistent estimators

for which the distribution of j~[Tn - 1"(8)] approaches the normal distribution with mean 0 and variance u 2(0). (iv) u 2 (0) is not less than u*\8) for all 0 in any open interval. fill

Remark The abbreviation BAN is sometimes replaced by CANE,

standing for consistent asymptotically normal efficient.

fill

The usefulness of this definition derives partially from tp.eorems proving the existence of BAN estimators and from the fact that ordinarily reasonable estimators are asymptotically normally distributed. It can be shown that for samples drawn from a normal density with mean /l and variance u 2 the sequence T: = (I/n)

L Xi = X n for
i:::: 1

n = I, 2, ...

is a BAN estimator of /l. In fact, the limiting distribution of J~(X n - /l) is normal with mean 0 and variance u 2 , and no other estimator can have smaller limiting variance in any interval of /l values. However, there are many other estimators for this problem which are also BAN estimators of /l, that is, esti mators with the same normal distribution in the limit. For example,

n
is a BAN estimator of /l. by (ii) of the definition.

1, 2, ... ,

BAN estimators are necessarily weakly consistent

PROPERTIES OF pOINT ESTIMATORS

297

3.4 Loss and Risk Functions

In Subsec. 3.2 we used mean-squared error of an estimator as a measure of the closeness of the estimator to 1"(8). Other measures are possible, for example, cK(l[1 T - 1"(0)1], called the mean absolute deviation. In order to exhibit and consider still other measures of closeness, we will borrow and rely on the language of decision theory. On the basis of an observed random sample from some density function, the statistician has to decide what to estimate 1"(8) to be. One might then caB the val ue of some estimator T = t( Xl' ... , X n) a decision and call the estimator itself a decision function since it tells us what decision to make. Now the estimate t of 1"(0) might be in error; if so, some measure of the severity of the error seems appropriate. The word" loss '" is used in place of "error," and" loss function" is used as a measure of the" error." A formal definition follows. Let t denote an estimate of 1"(0). The loss function, denoted by t(t; 0), is defined to be a real-valued function satisfying (i) t(t; 8) > 0 for all possible estimates t and all 8 in e and (ii) t(t; 0). = 0 for t = 1"(0). t(t; 0) equals the loss incurred if one estimates 1"(8) to be t when 0 is the true parameter value. IIII In a given estimation problem one would have to define an appropriate 10ssJutlcfion for the particular problem under study. It is a measure of the error and presumably would be greater for large error than for small error. We would want the loss to be small; or, stated another way, we want the error in estimation to be small, or we want the estimate to be close to what it is estimating. EXAMPLE 16 Several possible loss functions are: (i) tl(t; 8) = [t - 1"(0)]2. (ii) t 2 (t; 8) = 1t - 1"(8) I. (iii) (iv)
tl

Definition 11 Loss function Consider estimating 1"(0).

t (t. 8) =
3,

tit; 0) =

if 1t - 1"(8) I > e \0 if 1 t - 1"(0) I ~ e, where A > o. p(O) 1t - 1"(0)1' for P(O) > 0 and r > O.

fA

is called the squared-error loss function, and t 2 is called the absoluteerror loss function. Note that both t l and t 2 increase as the error t - 1"(8) increases in magnitude. t 3 says that you lose nothing if the estimate t is within e units of 1"(0) and otherwise you lose amount A. t 4 is a general IIII loss function that includes both t l and t 2 as special cases.

298

PARAMETRIC POINT ESTIMATION

vn

We assume now that an appropriate loss function has been defined for our estimation problem, and we think of the loss function as a measure of error or loss. Our object is to select an estimator T = t(Xl , .. , Xn) that makes this error or loss small. (Admittedly, we are not considering a very important, substantive problem by assuming that a suitable loss function is given. In general, selection of an appropriate loss function is not trivial.) The loss function in its first argument depends on the estimate t, and t is a value of the estimator T; that is, t = t(Xb " ., xn) Thus, our loss depends on the sample Xl' ... , X n We cannot hope to make the loss small for every possible sample, but we can try to make the loss small on the average. Hence, if we alter our objective of picking that estimator that makes the loss small to picking that estimator that makes the average loss small, we can remove the dependence of the loss on the sample Xl' "" X n This notion is embodied in the following definition.

Definition 12 Risk function For a given loss function t(; .), the risk function, denoted by [!llO), of an estimator T = t(Xl , .. , Xn) is defined to be
[!liO) = <if(J[t(T; 0)]. (10)

fIll
The risk function is the average loss. The expectation in Eq. (10) can be taken in two ways. For example, if the density f(x; 0) from which we sampled is a probability density function, then
C(J[t(T; 0)]

= C(J[t(t(Xh ... , =

X n); 0)]

f. .. J t(t(Xb""

,x n);

O)l~ f(Xi; 0) dXjo

We get

Or we can consider the random variable T and the density of T.

Co[t(T; 0)]

t( ; O)fT(t) dt,

where fT(t) is the density of the estimator T. averages out the values of Xl' . " , X n

In either case, the expectation

EXAMPLE 17 Consider the same loss functions given in Example 16. The corresponding risks are given by: (i) C(J[[T - T(O)j2], our familiar mean~squared error. (ii) <ifo[! T T(O) 1], the mean absolute error. (iii) A'Po[!T-T(O)! >8]. (iv) p(O)Co[! T - T(O)I''j. 1/11

SUFFICIENCY

299

Our object now is to select an estimator that makes the average loss (risk) small and ideally select an estimator that has the smallest risk. To help meet this objective, we use the concept of admissible estimators.

Definition 13 Admissible estimator For two estimators ~ = t 1(X1, ... , Xn) and T2 = t 2(X1, ... , X n), estimator t1 is defined to be a better estimator than t 2 if and only if
for all and for at least one
(J (J

in

e
e.

in

An estimator T = t(X1' ... , Xn) is defined to be admissible if and only if there is no better estimator. IIII In general, given two estimators t1 and t 2 neither is better than the other; that is, their respective risk functions as functions of (J, cross. We observed this same phenomenon when we studied the mean-squared error. Here, as there, there will not, in general, exist an estimator with uniformly smallest risk. The problem is the dependence of the risk function on (J. What we might do is average out (J, just as we average out the dependence on x., ... , Xn when going from the loss function to the risk function. The question then is: Just how should (J be averaged out? We will consider just this problem in Sec. 7 on the Bayes estimators. Another way of removing the dependence of the risk function on (J is to replace the risk function by its maximum value and compare estimators by looking at their respective maximum risks, naturally preferring that estimator with smallest maximum risk. Such an estimator is said to be minimax.

Definition 14 Minimax An estimator t* is defined to be a minimax estimator if and only if sup 9ft *(J) < sup 9ft<(J) for every estimator t. 1III
()
()

Minimax estimators will be discussed in Sec. 7.

SUFFICIENCY

Prior to continuing our pursuit of finding best estimators, we introduce the concept of sufficiency of statistics. In many of the estimation problems that we will encounter, we will be able to summarize the information in the sample

300

PARAMETRIC POINT ESTIMATION

VII

'-------

Xl' ... , Xn That is, we will be able to find some function of the sample that tells us just as much about () as the sample itself. Such a function would be sufficient for estimation purposes and accordingly is called a sufficient statistic. Sufficient statistics are of interest in themselves, as well as being useful in statistical inference problems such as estimation or testing of hypotheses. Because the concept of sufficiency is widely applicable, possibly the notion should have been isolated in a chapter by itself rather than buried in this chapter on estimation.

4.1

Sufficient Statistics

Let Xl, .. ' , Xn be a random sample from some density, say f(' ; (}). We defined a statistic to be a function of the sample; that is, a statistic is a function with domain the range of values that (Xl' "', Xn) can take on and counterdomain the real numbers. A statistic T = t(Xl , ... , Xn) is also a random variable; it condenses the n random variables Xl, X 2 , ... , Xn into a single random variable. Such condensing is appealing since we would rather work with unidimensional quantities than n-dimensional quantities. We shall be interested in seeing if we lost any "information" by this condensing process. The condensing can also be viewed another way. Let X denote the range of values that (Xl, ... , Xn) can assume. For example, if we sample from a Bernoulli distribution, then X is a collection of all n-dimensional vectors with components either 0 or 1; or if we sample from a normal distribution, then X is an n-dimensional euclidean space. Now a statistic induces or defines a partition of X. (Recall that a partition of X is a collection of mutually disjoint subsets of X whose union is X.) Let t(, ... , .) be the function corresponding to the statistic T = t(Xl , ... , Xn). The partition induced by t(, ... , .) is brought about as follows: Let to denote any value of the function t(, ... , .); that subset of X consisting of all those points (Xl' ... , xn) for which t(Xl, ... , xn) = to is one subset in the collection of subsets which the partition comprises; the other subsets are similarly formed by considering other values of t(, ... , '). For example, if a sample of size 3 is selected from a Bernoulli distribution, then X consists of eight points (0,0,0), (0,0,1), (0, 1,0), (1, 0,0), (0, 1, 1), (1, 0, 1), (1, 1,0), (1,1, 1). Let t(Xb X2, X3) = x] + x 2 + x 3 ; then t(, " .) takes on the values 0,1,2, and 3. The partition of X induced by t(, ., .), consists of the four subsets {CO, 0, O)}, {CO, 0,1), (0, 1,0), (1, 0, O)}, {CO, 1, 1), (1, 0, 1), (1, 1, O)}, and {(l, 1, 1)} corresponding, respectively, to the four values 0, 1, 2, and 3 of t(, " '). A statistic then is really a condensation of X. In the above example, if we use the statistic t(, " .), we have only four different values to worry about instead of the eight different points of X.

SUFFICIENCY

301

Several different statistics can induce the same partition. In fact, if t(., ... , .) is a statistic, then any one~to-one function of t has the same partition as t. In the example above t'(xl , Xl, X3) = 6(Xl + Xl + X3)2, Or even t"(x ,X , X ) = xi + X~ + x~, induces the same partition as t(Xl' X2' X3) = X +1 X ~ X 3. One of the reasons for using statistics is that they do condense and ff suc~ is our only reason for using a statistic, then any two statistics with the same partition are of the same utility. The important aspect of a statistic is the partition of that it induces, not the values that it assumes. A sufficient statistic is a particular kind of statistic. It is a statistic that condenses ~ in such a way that no "" information about 0" is lost. The only information about the parameter 0 in the density f( . ; 0) from which we sampled is contained in the sample Xl' ... , Xn; so, when we say that a statistic loses no information, we mean that it contains all the information about 0 that is contained in the sample. We enlphasize that the type of information of which we are speaking is that information about 0 contained in the sample given that we know the form of the density; that is, we know the function f('; .) in f(' ; 0), and the parameter 0 is the only unknown. We are not speaking of information in the sample that might be useful in checking the validity of our assumption that the density does indeed have formf('; '). Now we shall formalize the definition of a sufficient statistic; in fact, we shall give two definitions, namely, Definitions 15 and 16. It can be argued that the two definitions are equivalent, but we will not do it.

Definition 15 Sufficient statistic Let Xl' ... , Xn be a random sample

from the density f('; 0), where 0 may be a vector. A statistic S = O(Xb ... , Xn) is defined to be a sufficient statistic if and only if the con~ ditional distribution of Xl' ... , Xn given S = s does not depend on 0 for IIII any value s of s. Note that we use S = o(Xl' ... , X n), instead of T = t(Xl' ... , X n), to denote a sufficient statistic. Some care is required in interpreting the conditional distribution of Xl' ... , Xn given S = s, as Example 19 and the paragraph preceding it demonstrate. The definition says that a statistic S = o(Xl' ... , Xn) is sufficient if the conditional distribution of the sample given the value of the statistic does not depend on O. The idea is that if you know the val ue of the sufficient statistic, then the sample values themselves are not needed and can tell you nothing more about 0, and this is true since the distribution of the sample given the sufficient statistic does not depend on O. One cannot hope to learn anything about 0 by sampling from a distribution that does not depend on O.

302

PARAMETRIC POIN'l' ESTIMATION

VB

EXAMPLE 18 Let Xb X 2 , X3 be a sample of size 3 from the Bernoulli distribution. Consider the two statistics S = o(X., X 2 , X 3) = Xl + X 2 + X3 and T = t(XI , X 2 , X 3) = X I X 2 + XJ . We will show that 0(', " .) is sufficient and t(, " .) is not. This first column of Fig. 6 is X.

Values of S

Values of
T j
X l.X2,X3JS

X l,x2,X3I T

(0,0,0)

1-p l+p I-p 1 +2p p l+p p l+p p

(0,0, 1)

3
1

(0, 1,0)

3
1

(l, 0, 0)

3
1 3

(0,1,1)

1 + 2p
P

(1,0,1)

1 3 1 3 1

1 +2p p 1 +2p

(l, 1,0)

1
2

(1,1,1)

FIGURE 6

The conditional densities given in the last two columns are routinely calculated. For instance,

P[X I

= 0; X 2 = 1; X 3 = 0; S = 1] P[S = 1]
1.
p)2

(1 - p)P(1 - p)

G)p(l -

= 3'

sumClENCY

303

and

IXI. X2.X3IT==o(0, 1,01

o _P[X I
)-

=0;X z =I;X 3 =0;T=0] P[T = 0]

(1 _ p)zp
- (1 - p)3

+ 2(1

- p)zp

p
1- p

+ 2p

1+P

The conditional distribution of the sample given the values of S is independent of p; so Sis a sufficient statistic; however, the conditional distribution of the sample given the values of Tdepends on p; so Tis not sufficient. We might note that the statistic T provides a greater condensation of X than does S. A question that might be asked is: Is there a statistic which provides greater condensation of I than does S which is sufficient as well ? The answer is "no" and can be verified by trying all possible partitions of X consisting of three or fewer subsets. IIII In the case of sampling from a probability density function, the meaning of the term "the conditional distribution of Xl' ... , Xn given S = s that appears in Definition 15 may not be obvious since then P[S = s] = O. We can give two interpretations'. The first deals with the joint cumulative distribution function and uses Eq. (9) of Subsec. 3.3 in Chap. IV; that is, to show that S = o(XI' ... , Xn) is sufficient, one shows that P[XI Xl;"'; X" ~ xnl S = s] is independent of 0, where P[XI < Xl; ... ; X" < Xn 1S = s] is defined as in Eq. (9) of Chap. IV. The second interpretation is obtained if a one-to-one transformation of Xl' X z , ... , Xn to, say, S, Yz , ... , Yn is made, and then it is demonstrated that the density of Yz , ... , Yn given S = s is independent of O. If the distribution of Yz , ... , Yn given S = s is independent of 0, then the distribution of S, Yz , ... , Yn given S = s is independent of 0, and hence the distribution of Xl' X z , .. , Xn given S = s is independent of O. These two interpretations are illustrated in the following example.
H

EXAMPLE 19 Let Xl' "" X,. be a random sample from f(-; 0) = (J.I(); that is, X h ... , Xn is a random sample from a normal distribution with mean 0 and variance unity. In order to expedite calculations, we take n = 2. Let us argue that S = o(Xl' Xz) = Xl + X z is sufficient using the second interpretation above. The transformation of (Xl' Xz) to (S, Y z), where S = Xl + X z and Yz = X z - Xh is one-to-one; so it suffices to show that fY2 IS(Yz Is) is independent of O. Now

I Y2Is(Yzl s) = fr2.s(Yz,S) = fY2(yZ)fS<s) = f ( )

fs(s) Is(s)

Y2 Yz

304

PARAMETRIC POINT ESTIMATIoN

vn

(using the independence of Xl + X 2 and X2 - Xl that was proved in Theorem 8 of Chap. VI), but

since
Y2 ,..,., N(O, 2),

which is independent of O. The necessary calculations for the first interpretation above are less simple. We must show that P[XI < Xl; X 2 x21 S = s] is independent of O. According to Eq. (9) of Chap. IV,
P[XI < Xl; X 2 < x 2 1 S

= s] = lim P[XI < Xl;

h ...... o

X 2 < x21s - h < S < s + h].

Without loss of generality, assume that Xl < X2 . We have the following three cases to consider: (i) s < X., (ii) Xl < s < X2 , and (iii) X2 < s. P[XI < Xl; X 2 ::::;; Xli S = s] is clearly 0 (and hence independent of 0) for case (iii). Let us consider (i). [Case Oi) is similar.]
P[XI < Xl; X 2 < Xli S

= s]
Xl; X 2 < x 2 1s - h < S < s

= lim P[XI
h ...... o

+ h]

. 1 hm 2h P[XI
h-+o

Xl; Xl < Xl;S - h < S < s + h]

S

---------------------------------lim Zl P[s - h < S < h ...... o h

+ h]

-----------------------------------fs(s)
Note that (see Fig. 7)

h ...... O

. 1 hm Zh P[XI < Xl; X 2

X2; s - h < S < s

+ h]

1 lim h ...... O

Zh

J f
Xl

s+h-Il

!Xl(U)!X2(V) dv du,

S-h-X2

s-h-Il

SUFFICIENCY

305

v
___ X2 _ _ _ _ _ _ _ _ _ _ _

+ __ _
I

u+
-41---_u

FIGURE 7

and hence

Finally, then,
P[XI <
Xl;

X 2 < x21S = s]

J:~X2 !Xt(U)!X2(S !S(S)

- U) 2

du

J:~x2(1/2n)e -![u - 2uO+6 2+(s-u)2_ 2(s-u)6+6 2] du

(1/ J 4n)e -![(s- 26)/"'2]2

(I/J~) J:~X2e-![U2+(S-U)2] du

(1 IJ'Z)e- !(s2/2)
which is independent of O.

11//

306

pARAMETRIC POINT ESTIMATIoN

VII

Definition 15 of a sufficient statistic is not very workable. First, it does not tell us which statistic is likely to be sufficient, and, second, it requires us to derive a conditional distribution which may not be easy, especially for continuous random variables. In Subsec. 4.2 below, we will present a criterion that may aid us in finding sufficient statistics. Although we will not so argue, the following definition is equivalent to Definition 15.

Definition 16 Sufficient statistic Let Xb ... , Xn be a random sample from the density f(; e). A statistic S = d(Xl' ... , Xn) is defined to be a
sufficient statistic if and only if the conditional distribution of T given S does not depend on e for any statistic T = t(Xb ... , Xn). IIII

Definition 16 is particularly useful in showing that a particular statistic is not sufficient. For instance, to prove that a statistic T' = t'(Xl , ... , Xn) is not sufficient, one needs only to find another statistic T = t(Xb ... , Xn) for which the conditional distribution of T given T' depends on e. For some problems, no single sufficient statistic exists. However, there will always exist jointly sufficient statistics.

Definition 17 Jointly sufficient statistics Let Xl' "', Xn be a random

sample from the density f(' ; e). The statistics S1, ... , Sr are defined to be jointly sufficient if and only if the conditional distribution of Xb ... , Xn given Sl = Sl' ... , Sr = Sr does not depend on e. IIII The sample Xl, ... , Xn itself is always jointly sufficient since the conditional distribution of the sampJe given the sample does not depend on e. Also, If the the order statistics Y l , ... , Yn are jointly sufficient for random sampling. order statistics are given, say, by (Yl = Yl, ... , Yn = Yn), then the only values that can be taken on by (Xl' ... , Xn) are the permutations of Yl, ... , Yn' Since the sampling is random, each of the n! permutations is equally likely. So, given the values of the order statistics the probability that the sample equals a particular permutation of these given values of the order statistics is lIn!, which is independent of e. (Sufficiency of the order statistics also follows from Theorem 5 belOW.) If we recall that the important aspect of a statistic or set of statistics is the partition of x that it induces, and not the values that it takes on, then the validity of the following theorem is evident.

SUFFICIENCY

307

Theorem 3 If SI = 01(X1, ... , Xli)' " ., S,. = o,,(Xb ... , Xli) is a set of jointly sufficient statistics, then any set of one-to-one functions, or transformations, of 8 1 , .. , S,. is also jointly sufficient. I111
For example, if L Xi and L X; are jointly sufficient, then X and L (Xi - X)2 = L xl -" nX2 are also jointly sufficient. Note, however, that X 2 and L (Xi - X)2 may not be jointly sufficient since they are not one~to-one functions of L Xi and L X;. We note again that the parameter that appears in any of the above three definitions of sufficient statistics can be a vector.

4.2

Factorization Criterion

The concept of sufficiency of statistics was defined in Definitions 15 to 17 above. In many cases, a relatively easy criterion for examining a statistic or set of statistics for sufficiency has been developed. This is given in the next two theorems, the proofs of which are omitted.

Theorem 4 Factorization theorem (single sufficient statistic) Let X h

X 2 ' ... , Xli be a random sample of size n from the density f(; e), where the parameter e may be a vector. A statistic S = o(Xl , ... , Xli) is sufficient
II

ifand only if the joint density of X b ... , Xli' which is nf(x i ; e), factors as
i
1

fXt, ''', Xn(X h ... , Xli; e) = g(O(Xl' ... , Xli); e)h(Xb ... , Xli)

= g(S; e)h(Xl' ... , Xli)'

(II)

where the function h(Xl' ... , Xli) is nonnegative and does not involve the parameter e and the function g(O(Xh ... , Xli); e) is nonnegative and depends on Xl, ... , Xli only through the function 0(', ... , '). IIII

Theorem 5 Factorization theorem (jointly sufficient statistics) Let Xb

X 2 , ... , Xli be a random sample of size n from the density f( ; (J), where the parameter e may be a vector. A set of statistics Sl = 0l(Xl , ... , Xn), .. , S,. = o,,(Xh ... , Xli) is jointly sufficient if and only if the joint density of X h .. , Xli can be factored as fxt, ... ,Xn(Xl' ... , Xli; 0)
= g(Ol(Xb ... , Xli)' .. , o,,(Xb ... ,

xJ; (J)h(Xh ., ., Xli)

(12)

= g(Sb ... , s,.; (})h(Xl' ... , Xli)'

308

PARAMETRIC POINT ESTIMATION

VII

where the function h(Xl' ... , xn) is nonnegative and does not involve the parameter fJ and the function g(Sb ... , sr; fJ) is nonnegative and depends on Xl' ... , Xn only through the functions .11(', ... , .), .. " dr(", .. " .). II/I Note that, according to Theorem 3, there are many possible sets of sufficient statistics. The above two theorems give us a relatively easy method for judging whether a certain statistic is sufficient or a set of statistics is jointly sufficient. However, the method is not the complete answer since a particular statistic may be sufficient yet the user may not be clever enough to factor the joint density as in Eq. (11) or (12). The theorems may also be useful in discovering sufficient statistics. Actually, the result of either of the above factorization theorems is intuitively evident if one notes the following: If the joint density factors as indicated in, say, Eq. (12), then the likelihood function is proportional to g( Sl' ... , Sr; fJ), which depends on the observations Xb .. " Xn only through .11' .. " dr [the likelihood function is viewed as a function of fJ, So h(xb ... , xn) is just a proportionality constant], which means that the information about fJ that the likelihood function contains is embodied in the statistics .11(', ... , .),
, d r(', , ').

Before giving severa~ examples, we remark that the function h(, "', .) appearing in either Eq. (11) or (12) may be constant.

EXAMPLE 20 Let Xb ... , Xn be a random sample from the Bernoul1i density with parameter fJ; that is, and Then

,=1

.n !(Xi; fJ) = n fJxi(I i=l

e)l-xiJ {O, l}(Xi)

fJl:Xi(1 - fJt-l: Xi

i=l

nl{O,l}(Xi)'
n
i= 1

If we take fJl:Xi(I -

fJt -l:x,

as g(d(Xl' "', x n); fJ) and

I{O,l}(Xj)

as

h(Xl' .. " xn) and set d(Xl' ... , xn) = Xi' then the joint density of Xb ... , Xn factors as in Eq. (11), indicating that S = d( Xl' ... , Xn) = Xi is a sufficient statistic. IIII

SUFFICIENCY

309

EXAMPLE 21 Let Xl' ... , XII be a random sample from the normal density with mean Jl and variance unity. Here the parameter is denoted by Jl instead of 8. The joint density is given by

Ix

10

X n( Xl'

. "

x.. ; Jl) =
,.

i= 1

n p.l(Xj)

i=l

Ii J2n 1 exp [- ~ (Xi - Jl)2] 2

= (2n)II/2 exp =

[1 - 2i~l
II
r

(Xi -

Jl)

2]
L: x, + nJl2)]
exp (- 2 ~ i...J '\~ x~). I

(2,,~nI2 exp [ - ~ (L: xl -

2Jl

= (2nt/2 1 exp

(II" X. i...J ,

~ 2 Jl2)

Ifwetakeh(xl"'" XII) = [l/(2n)II/2] exp (-lIxf) andg(O(xl' ... , XII); Jl) = exp J.L I Xi - (nI2)Jl2], then the joint density has been factored as in Eq. (11) with O(Xl' ... , XII) = I Xi; hence I Xl is a sufficient statistic. (Recall that X II is also sufficient since any one-to-one function of a sufficient statistic is also sufficient.) II/I

EXAMPLE 22 Let Xl' . , Xn be a random sample from the normal density p, a2( . ). Here the parameter () is a vector of two components; that is, 8 = (j.t, u). The joint density of Xl, ... , XII is given by

iQ <P
II

p ,a2(X i )

1 [1 (X' - Jl) 2] I\ J2nu exp - 2

n
I

= (2:),,12 ,,-0 exp [ _

~ L: (XI ~ Jl) 2]
2jl

= (2,,~/2 ,,-0 exp [ - 2!2 (L: xl -

L: XI + nJl2)] ;

so the joint density itself depends on the observations Xl' ... , XII only through the statistics 01(Xh ... , XII) = L Xi and 02(X I , ... , XII) = I x;; that is, the joint density is factored as in Eq. (12) with h(Xb ... , xn) = 1. Hence, L Xi and I Xl are jointly sufficient. It can be shown that X It and S2 = [l/(n - 1)] I (Xi - X)2 are one-to-one functions of I Xi and L xl; so XII and S2 are also jointly sufficient. 11//

310

PARAMETRIC POINT ESTIMATION

VII

EXAMPLE 23 Let Xl' ... , X,. be a random sample from a uniform distribution over the interval [01 , O The joint density of Xh .. , X,. is given by 2 ],

IXI ..... X/Xl' . , x,.; Oh ( 2 ) =

i= 1

n0
2 -

,.

1
2 -

0 I[fJt.fJ2](Xi)
1

1
= (0

0 ),.
1

i:;l

n l[fJ

fJ2](Xj)

1
=

(0

_ ( 2

)" l[fJt.1n](Yl)1[;Pl,fJ2](Y")' 1

where and

y,.

max [Xb ... , X,.].

The joint density itself depends on Xh " ' , x,. only through Y1 and y,.; hence it factors as in Eq. (12) with h(X1' ... , X,.) = 1. The statistics Y 1 and Y,. are jointly sufficient. Note that if we take 01 = 0 and O = 0 + 1, 2 then Y1 and Y,. are still jointly sufficient. However, if we take 01 = 0 and O 2 = 0, then our factorization can be expressed as

0" 1[0, fJ](Y,.)I [0, 1n](Y 1)'

1[0. Yn](Yl),

Taking g(a(xb ... , X,.); 0) = (1/8")1[0. fJ](yn) and h(Xb ... , xn) = we see that Y,. alone is sufficient.

IIII

The factorization criterion of Eqs. (11) and (12) is primarily useful in showing that a statistic or set of statistics is sufficient. It is not useful in proving that a statistic or set of statistics is not sufficient. The fact that we cannot factor the joint density does not mean that it cannot be factored; it could be that we are just not able to find a correct factorization. If we go back and look through our examples on maximum-likelihood estimators (see Examples 5 to 8), we will see that all the maximum-likelihood estimators that appear there depend on the sample Xl' ... , X,. through sufficient statistics. This is not something that is characteristic of the relatively simple examples we had given but something that is true in general.

Theorem 6 A maximum-likelihood estimator or set of maximumlikelihood estimators depends on the sample through any set of jointly sufficient statistics.

SUFFICIENCY

311

If Sl = 01(Xb ... , X n), "" Sk = 0k(Xb ... , Xn) are jointly sufficient, then the likelihood function can be written as
PROOF

=
=

=1

n
n

!(Xi; 0)
,

g(ol(XI ,

x n),

0k(XI , .,., x n); O)h(XI' ... , x n)

As a function of 0, L(O; Xl' ... , xn) will have its maximum at the same place that g(SI' .. , Sk; 0) has its maximum, but the place where 9 attains its maximum can depend on Xl' ... , Xn only through Sb ... , Sk since 9
~~.

HH

We might note that method-of-moment estimators may not be functions of sufficient statistics. See Examples 4 and 23.

4.3

Minimal Sufficient Statistics

When we introduced the concept of sufficiency, we said that our objective was to condense the data without losing any information about the parameter. We have seen that there is more than one set of sufficient statistics. For example, in sampling from a normal distribution with both the mean and variance unknown, we have noted three sets of jointly sufficient statistics, namely, the sample 2 Xl' ... , Xn itself, the order statistics Y We naturally I , .. , Yn , and X and 8 . prefer the joi ntly sufficient set X and 8 2 since they condense the data more than either of the other two. (Note that the order statistics do condense the data.) The question that we might ask is: Does there exist a set of sufficient statistics that condenses the data more than X and 8 2 ? The answer is that there does not, but we will not develop the necessary tools to establish this answer. The notion that we are alluding to is that of a minimum set of sufficient statistics, which we label minimal sufficient statistics. We noted earlier that corresponding to any statistic is the partition of ~ that it induces. The same is true of a set of statistics; a set of statistics induces a partition of ~. Loosely speaking, the condensation of the data that a statistic or set of statistics exhibits can be measured by the number of subsets in the partition induced by that statistic or set of statistics. If a set of statistics has fewer subsets in its induced partition than does the induced partition of another set of statistics, then we say that the first statistic condenses the data more than the latter. Still loosely speaking, a minimal sufficient set of statistics is then a sufficient set of statistics that has fewer subsets in its partition than the induced

312

pARAMETRIC POINT ESTIMATION

VII

partition of any other set of sufficient statistics. So a set of sufficient statistics is minimal if no other set of sufficient statistics condenses the data more. A formal definition is the following. Definition 18 Minimal sufficient statistic A set of jointly sufficient statistics is defined to be minimal sufficient if and only if it is a function of every other set of sufficient statistics. IIII Like many definitions, Definition 18 is of little use in finding minimal sufficient statistics. A technique for finding minimal sufficient statistics has been devised by Lehmann and Scheffe [19], but we will not present it. If the joint density is properly factored, the factorization criterion will give us minimal sufficient statistics. All the sets of sufficient statistics found in Examples 20 to 23 are minimal.

4.4 Exponential Family

Many of the parametric families of densities that we have considered are members of what is called the exponential class, or exponentialfamity, not to be confused with the negative exponential family of densities which is a special case. Definition 19 Exponential family of densities A one-parameter family (lJ is unidimensional) of densities f( ; lJ) that can be expressed as
f(x; lJ) = a(lJ)b(x) exp [c(lJ)d(x)] (13)

for - 00 < x < 00, for all lJ E e, and for a suitable choice of functions a(')' b('), c(), and d() is defined to belong to the exponential family or exponential class. IIII EXAMPLE 24 If f(x; lJ) = 8e- ex/(0. ooix), then f(x; lJ) belongs to the exponential family for a(fJ) = lJ, b(x) = /(0. tt:l)(x), c(lJ) = - 8, and d(x) = x in Eq. (13). IIJI EXAMPLE 25
Iff(x; lJ) =f(x; A) is the Poisson density, then

e-A.1x
f(x; A) =

x.

,/{o.l ... i x )

SUFFICIENCY

313

In Eq. (13), we can take a(A) = e- , b(x) = (1Ix!)I{o.l ... }(x), C(A) = log A, and d(x) = x; so f(x; A) belongs to the exponential family. /III

Remark If f(x; 0)

a(O)b(x) exp [c(J)d(x)], then

,U f

(x,; 6) =

a'(6{Jj, b(X,)] exp [c(6),t,d(x,)].

IIII

and hence by the factorization criterion L d(Xi) is a sufficient statistic.

. The above remark shows that, under random sampling, if a density belongs to the one-parameter exponential family, then there is a sufficient statistic. In fact, it can be shown that the sufficient statistic so obtained is minimal. The one-parameter exponential family can be generalized to the k-parameter exponential family.

Definition 20 k-parameter exponential family A family of densities

f( . ;

(h, ... , Ok) that can be expressed as

j=l

f(x; (}l' ... , (}k) = a({}l, ... , (}k)b(x) exp

L Ci{}l' ... , (}k)d/x)

(14)

for a suitable choice of functions a(, ... , .), b(), ci, ... , .), and dj (), j = 1, ... , k, is defined to belong to the exponential family. IIII In Definition 20, note that the number of terms in the sum of the exponent is k, which is also the dimension of the parameter.

EXAMPLE 26 Iff(x; {}l, (}2) = cP". 2 (x) , where (Ol' (}2) belongs to the exponential family.
(I

(Ji, a), thenf(x; {}1I (}2)

1 exp (1 = fo - -2 . Ji 2na a
2

2 )

exp

(1 2a

-2

x2

+ ..!!..x 2
a

Take a(p, a) = (1/J2na) exp (--!- . Ji 2I( 2), b(x) = 1, Cl(P, a) = -1/2a2, 2 C2(P, a) = Jila , dl(x) = x 2, and d2(x) = x to show that cP",(l2(X) can be expressed as in Eq. (14). IIII

314

PARAMETRIC POINT ESTIMATION

VII

EXAMPLE 27 If

then
J(X;

1, (

1 2) = B(Ol' 0l) 1(0,1) (x) exp [(Ot - 1) log x

+ (0 2 -

I) log (1 - x)];
= =

so J(x;

I/B(Ol, ( 2 ), b(x) = 1(0, l)(X), log x, and d2(x) = log (1 - x).

1,

( 2)

belongs to the exponential family with a(Ol' ( 2 )

Ct (0)

= 01 - 1, ciO) = O2 - I, d1(x)

IIII

Remark Iff(x; Of, ... , Ok) = a(Ol, ... , 0k)b(X) exp

then, under random sampling,

j=l

C/Oh ... , Ok)d/x),

i= 1

n f (Xi;
II

1, . . .,

k)
I
II

and so by the factorization criterion jointly sufficient statistics. sufficient statistics.

:=1

d1(Xi ),

"',

dk(Xi ) is a set of

d1(X,), ... ,

i=1

dk(Xi) are in fact minimal

1I11

EXAMPLE 28

From Example 27, we see that

II

log Xi and

II

log (1 - Xl)

i=l

1==1

are jointly minimal sufficient when sampling from a beta density.

1111

Our main use of the exponential family will not be in finding sufficient statistics, but it will be in showing that the sufficient statistics are complete, a concept that is useful in obtaining" best" estimators. This concept will be defined in Sec. 5. Lest one get the impression that all parametric families belong to the exponential family, we remark that a family of uniform densities does not belong to the exponential family. In fact, any family of densities for which the range of the values where the density is nonnegative depends on the parameter does not belong to the exponential class.

s
5 UNBIASED ESTIMATION

UNBIASED ESTIMATIoN

315

Since estimators with uniformly minimum mean-squared error rarely exist, a reasonable procedure is to restrict the class of estimating functions and look for estimators with uniformly minimum mean-squared error within the restricted class. One way of restricting the class of estimating functions would be to consider only unbiased estimators and then among the class of unbiased estimators search for an estimator with minimum mean-squared error. Consideration of unbiased estimators and the problem of finding one with uniformly minimum mean-squared error are to be the subjects of this section. According to Eq. (9) the mean-squared error of an estimator T of 1'(6) can be written as t!8[[T - 1'(6)]2] = var6 [T] + {1'(0) - t!8[T]}2, and if T is an unbiased estimator of 1'(0), then t!8[T] = 1'(8), and so t!6[[T - 1'(0)]2] = varo [T]. Hence, seeking an estimator with uniformly minimum mean-squared error among unbiased estimators is tantamount to seeking an estimator with uniformly minimum variance among unbiased estimators.

Definition 21 Uniformly minimum-variance unbiased estimator (UMVUE)

Let Xl' ... , Xn be a random sample from f(; 0). An estimator T* = t*(XI' "', XJ of 1'(0) is defined to be a uniformly minimum-variance unbiased estimator of 1'(0) if and only if (i) t!8[T*] = 1'(0), that is, T* is unbiased, and (ii) var8 [T*] < var6 [T] for any other estimator T = t(XI ,., XII) of 1'(0) which satisfies t!o[T] = 1'(0). IIII In Subsec. 5.1 below we will derive a lower bound for the variance of unbiased estimators and show how it can sometimes be useful in finding an UMVUE. In SUbsec. 5.2 we will introduce the concept of completeness and show how it in conjunction with sufficiency can sometimes be used to find an UMVUE.

5.1

Lower Bound for Variance

Let X b ., Xn be a random sample from/(; (J), where 6 belongs to e. Assume that e is a subset of the real line. Let T = t(Xh ... , Xn) be an unbiased estimator of 1'(0). We will consider the case where f(; 0) is a probability density function; the development for discrete density functions is analogous. We make the following assumptions, caned regularity conditions:
(i)

a log f(x;O) ao

exists for all x and all O.

316

PAltAMETRlC POINT ESTIMATIoN

vn

(ii)

! rf ,a !(x,;
=
8 f . .. ae
=

0)

dx, ... dx.

r. f:O,a!(X,;9)dX, "dx.
i= 1

(] 'I']')

ft{ Xl' , XII )IT" f( Xi; e) dXl , . dXn

IJ) dx,

r. f t(x" ... , x.) ! ,D/(X,;

[[:0 logj(X; 0)],] <
00 for

... dx.
e.

(iv) 0 < S.

all 9 in

Theorem 7

Cramer-Rao inequality Under assumptions (i) to (iv)

above vare [T]

n8'[[:9

(15)

10gj(X;

/l)] ']

where T = t{Xl' ... , Xn) is an unbiased estimator of ,(e). Equality prevails in Eq, (15) if and only if there exists a function, say K(e, n), such that

8 f 8e log!(xi; e) = K(e, n)[t(xh .. "

n

Xn) - ,(e)],

(16)

Equation (15) is called the Cramer-Rao inequality, and the right-hand side is called the Cramer-Rao lower bound for the variance of unbiased estimators of ,(e).
PROOF

8 ,(e) = 8e 8 ,'(e) = 8e
=

" f(Xj; e) dX1 , . dXII ,IJ J ..... f t {X1' .. , xn) 80 ~ [rlf{Xi; e)] dX1 ... dXII i=1
. , XII)
1

f . . . Jt{Xl"

- T(O) :0

r.

f ... f,U [f(X,; 0) dx;l

:9

f t(X" .... x.)

- 9)

r. f

:9 [.D/(X,; 9)] dx, ." dx.

[,:0, !(x,; 9)] dx, ... dx.

UNBIASED ESTIMAnON

317

r.

I[t(x" ... , x.,} - 1(/1)]

:0 Lr"l fix,; 0] <lx, ... <lx,

I' I[t(x" .. , x,) - -r(6)] [:6 IOg.U j(x,; 0)]

= 4'. [[t(X " .. , X.) - -r(/I)] [:6 log tV(X,; 0)]].

Now by the Cauchy-Schwarz i'nequality

[1'(0)]2 < S.[[t(X " ... , X,) - 1(0)]2]S.

[[:0 log ,:0. f(X,; 6)] 'l

or

but

8'[[a~ log,:o. f(X,; 6)rJ = s.[[,t, a~ log/(X,; 6)n

= =

~ ~ 8.[[:6 logf(X,; 6)] [:6 10gf(X); 0)]]

n.r.[[:6 10gf(X; 0)] 2],

using the independence of Xi and Xj and noting that

S.[:o!Og/(X; /I)] =
=

I [! log/(x; /I)]i(x; 0) dx

aof(x; (J) dx

= 00

ff(x; 0) dx

= ao (1) = O.

The inequality in the Cauchy-Schwarz inequality becomes an equality if and only if one function is proportional to the other; in our case this requires that :0 log or that there exists K

,:0. /(x,; 0) be proportional to I(x" ... , x.) ... , xJ - 1(0)].

1(0)

= K(O, n) such that

1111

;o!Og Ux /(x,; 0)] = K(O, n)[t(x"

318

pARAMETRIC POINT ESTIMATION

vn

The regularity conditions, which were stated for probability density functions, can be modified for discrete density functions, leaving the statement of the theorem unchanged. The theorem has two uses: First, it gives a lower bound for the variance of unbiased estimators. An experimenter using an unbiased estimator whose variance was close to the Cramer-Rao lower bound would know that he was using a good unbiased estimator. Second, if an unbiased estimator whose variance coincides with the Cramer-Rao lower bound can be found, then this estimator is an UMVUE. Equation (16) aids in finding an estimator whose variance coincides with the Cramer-Rao lower bound. In fact, if there exists a T* = t*(Xl' ... , X,.) such that
~ ae logf(xi; e) = K(e, n)[t*(xl' ... ,

,. 0

xJ -

-r*(e)]

for some functions K(e, n) and -r*(e), then T* is an UMVUE of -r*(e). EXAMPLE 29 Let Xl' ... , X,. be a random sample from f(x; e) = ee- 9x/(o.OO)(x). Take -r(e) = e. It can be shown that the regularity conditions are satisfied. -r' (e) = 1 ; hence

Note that oe logf(x; e) = ae (log e - ex)

lIe - x, and so

Hence, the Cramer-Rao lower bound for the variance of unbiased estimators of e is given by

Similarly the Cramer-Rao lower bound for the variance of unbiased estimators of -r(e) = lIe is given by

s
The left..hand side of Eq. (16) is

UNBIASED ESTIMATION

319

t
1

!-.log/(x,; 00

e) =

t
1

!..(lOg e 00

eXi) = f. (-e - Xi) = -n(xn - ~), 1

1

By taking K(O, n) = -n and utilizing the result of Eq. (16), we see that X n is an UMVUE of lIe since its variance coincides with the Cramer-Rao lower bound. IIII

EXAMPLE 30 Let Xl' ... , Xn be a random sample fromJ(x; 0) =J(x; A) = e- AA:1I: Ix! for x = 0, I, 2, ....

A). 10gJ(x; A) = a). log x!

=

e-.lAx

OA (-A

+ x log A -log x!) =

-I

+ l'

Therefore

... [[~ logf(X;

l)n

=8
1
=

[(~ -1)']

= ;, 8[(X -l)']
1 1 A= ~ ,

;.:z var [X] =

).2

and so the denominator of the Cramer-Rao lower bound is n/A. Now, if 1'(A) = e- A = P[X = 0], then the Cramer-Rao lower bound for the vanance of unbiased estimators of 1'(.1) = e- A is given by var [T] > n 2A k- /n. Note that T = (l/n) I I{o)( Xi) is an unbiased estimator of 1'(.1)
i= 1

e- A since 8[T] = (lIn)

'L 8[I{o}(Xj)] = (l/n) Ie-A = e- A.

i=l 1==1

I{olX;) = 1

if Xi = 0, and I(oj(Xi) = 0 otherwise; so 8[I{o}(Xi )] = 1 ' P[Xi = 0] + 0 . P[Xj #- 0] = e- A T is the proportion of observations in the sample ), as compared to the that are equal to O. var [T] (l/n)e-A(l - e- A Cramer-Rao lower bound, which is (1/n)k-2A. Note that ) > (l/n)J.e- 2A, (l/n)e-A(I - e- A as it should be. An UMVUE of 1'(A) = e- A is found in Example 34. We note that I (O/OA) logJ(x,; A) = I (-1 + XdA) = (njA)(x - A); hence, X is the UMVUE of A by Eq. (16). ///1

320

pARAMETRIC POINT ESTIMATION

vn

In general, the Cramer-Rao lower bound is not an attainable lower bound; that is, there often exists a lower bound for variance that is greater than the Cramer-Rao lower bound. We will see several such examples in Subsec. 5.2 below. We win see that an UMVUE can exist whose variance does not coincide with the Cramer-Rao lower bound. We conclude this subsection with several remarks, the statements of which are not necessarily mathematically precise. All the same, the remarks are important and do relate some earlier concepts to the Cramer-Rao lower bound.

Remark' Under certain assumptions involving the existence of second

derivatives and the validity of interchanging the order of certain differentiations and integrations,

0".[ [:0 iogf( X; O)rJ = -of. [::2 iogf( X; IJ)

IIII

This remark is computationally useful if the first expectation is more difficult to obtain than the second. The proof is left as an exercise.

Remark If the maximum-likelihood estimate ofB, say (j

is given by a solution to the equation

S(Xl' "', x n),

aB log L(B;

Xb ... ,

Xn)

= aB log

Jl !(X
n

j ;

B)

= 0,

and if T* = t*( Xl, ... , Xn) is an unbiased estimator of T*(B) whose variance coincides with the Cramer-Rao lower bound, then t*(Xb' .. , xn) = T*(S( Xl, ... , Xn.
PROOF

o=

a
aB log

iQ !(x

j ;

B)

(1-9 =

K(B, n)[t*(xl' ... , Xn) - T*(B)]

(lo:=(j

by Eq. (16) and the definition of

e.

IIII

This remark tells us that under the conditions of the remark a maximumlikelihood estimator is an UMVUE!

t*{Xl , ... , Xn) is an unbiased estimator of some T*(B) whose variance coincides with the Cramer-Rao lower bound, then f(; B) is a member of the exponential class; and, conversely, if f( ; B) is a member of the exponential class, then there exists an unbiased estimator, say T*, of some function, say T*(B), whose variance coincides with the Cramer-Rao lower bound. 1///
=

Remark If T*

UNBIASED ESTIMATION

321

We win omit the proof of this remark. It relates the Cramer-Rao lower bound to the exponential family; in fact, it tells us that we will be able to find an estimator whose variance coincides with the Cramer-Rao lower bound if and only if the density from which we are sampling is a member of the exponential class. Although the remark does not explicitly so state, the following is true: There is essentially only one function (one function and then any linear function of the one function) of the parameter for which there exists an unbiased estimator whose variance coincides with the Cramer-Rao lower bound. So, what this remark and the comments following it really tell is: The CramerRao lower bound is of limited use in finding UMVUEs. It is useful only if we sample from a member of the one-parameter exponential family, and even then it is useful in finding the UMVUE of only one function of the parameter. Hence, it behooves us to search for other techniques for finding UMVUEs, and that is what we do in the next subsection. 5.2 Sufficiency and Completeness In this subsection we will continue our search for UMVUEs. Our first result will show how sufficiency aids in this search. Loosely speaking, an unbiased estimator which is a function of sufficient statistics has smaller variance than an unbiased estimator which is not based on sufficient statistics. In fact, let f( . ; 0) be the density from which we can sample, and suppose that we want to estimate t(O). Let us assume that T = t(XI' " ., Xn) is an unbiased estimator of t(O) and that S = Q(XI , .. , Xn) is a sufficient statistic. It can be shown that another unbiased estimator, denoted by T', can be derived from T such that (i) T' is a function of the sufficient statistic Sand (ii) T' is an unbiased estimator of t(8) with variance less than or equal to the variance of T. Therefore, in our search for UMVUEs we need to consider only unbiased estimators that are functions of sufficient statistics. We shall formalize these ideas in the following theorem. Theorem 8 Rao-Blackwell Let Xh ... , Xn be a random sample from the density f( ; 8), and let SI = Ql(X1 , ... , X) n" , S k = Vk (X1, .. , X) n be a set of jointly sufficient statistics. Let the statistic T = t(Xb ... , X,,) be an unbiased estimator of t(8). Define T' by T' = 8[TIS1' ... , Sk]' Then,
A

(i) T' is a statistic, and it is a function of the sufficient statistics Sb ... , Sk' Write T' = t'(Sb ... , Sk). (ii) 8 o[T'] = t(8); that is, T' is an unbiased estimator of t(8). (iii) varo [.T'] < varo [T] for every 8, and varo [T'] < vare [T] for some 8 unless T IS equal to T' with probability 1.

322

PARAMETRIC POINT ESTIMATION

VII

(i) SI' . , Sk are sufficient statistics; so the conditional distribution of any statistic, in particular the statistic T, given Sf, ... , Sk is independent of B; hence T' = G[TI Sf, ... , Sk] is independent of B, and so T' is a statistic which is obviously a function of Sf, ... , Sk' (ii) G9[T'] = G9[G[TISl' ... , Sd] = G9[T] = r(B) [using Eq. (26) of Chap. IV]. (iii) We can write
PROOF

var9[T] = G9[tT-G o[T'])2] =G6[(T- T' +T' - 8 6[T,])2] = tf6[(T - T,)2] + 2G6[(T - T')(T' - tf6[T'])] + var6 [Tl But
8 6[(T - T')(T' - 8 6[T'])] = 8 6[tf[(T - T' )(T' - tfll[T'])! Sf, .. " Sk]]'

and
G[(T - T')(T' - G6[T']) ISI = Sl; ... ; Sk = sd = {t'(Sl' .. " Sk) - 8 6[T']}G[(T - T')I SI = SI; ... ; Sk
= {t'(sp ... , Sk) - 8 o[T']}(t9'[T lSI = SI; ... ; Sk = SkJ

= sd

- 8[T' lSI = SI; ... ; Sk = Sk])

= {t'(S1' ... , Sk) - tB'6[T']}[t'(Sj, ... , Sk) - t'(S1, ... , Sk)]

=0,

and therefore var9 [11 = Go[(T - T')2]

+ var6 [Tf] > varo [T'].

Note that varo [T] > varo [T'] unless Tequals T' with probability l.

II/I

For many applications (particularly where the density involved has only one unknown parameter) there will exist a single sufficient statistic, say S = O(Xl' ... , X n), which would then be used in place of the jointly sufficient set of statistics SI' ... , Sk' What the theorem says is that, given an unbiased estimator, another unbiased estimator that is a function of sufficient statistics can be derived and it will not have larger variance. To find the derived statistic, the calculation of a conditional expectation, which mayor may not be easy, is required.

EXAMPLE 31 Let Xl' ... , X,. be a random sample from the Bernoulli density I(x; B) = (]x(I - B)1-;;c for x = 0 or 1. Xl is an unbiased estimator of r(B) = B. We use Xl as T = t(Xl , ... , XII) in the above theorem. Xi is a sufficient statistic; so we use S = I Xi as our set (of one element) of

UNBIASED ESTIMATION

323

sufficient statistics. According to the above theorem T' = t9'[T 1 S1 = 8[X 1L X.] is an unbiased estimator of 0 with no larger variance than T =1 Xl- Let us evaluate t9'[T 1S1. We first find the conditional distribution of Xl given L Xi = S. Xl takes on at most the two values 0 and 1.

p[X 1 = OIL Xi = s] =

P[x,

=O;.tXi=
p==l ]

[n P LX' =s 1
i=

$]

P[tXi
t= 1

=s]

P[X 1

= 1]

p[ f Xi= -1]
S
i=2

P[it Xi

s]

- --(-:)-es-(1-0-r--- s

(n - 1 1)8
S-

s - 1(1

_ 0t- 1 -

s+ 1
S

We note in passing that the conditional distribution of Xl given L Xl = S is independent of 8, as it should be. Also, we could have derived the conditional distribution with much less effort by asking: Given that you have observed s successes in n trials, what is the probability that the first trial resulted in a success? This probability is sIn. (See Example 28
in Chap. I.)

324

PAltAMETRlC POINT ESTIMATION

vu

hence,
II

T' =

L
i=l

Xi

The variance of Xl is 9(1 - 8), and the variance of T' is 9{ I - 8)ln; so for n > 1 the variance of T' is actually smaller than the variance of
T

Xl'

//1/

Before leaving Theorem 8, two comments are appropriate: First, if the unbiased estimator T is already a function of only Sl' ... , S'k, then the derived statistic T' will be identical to T, and hence no improvement in variance can be expected. Second, although the set of jointly sufficient statistics is an arbitrary set, in practice one would naturally use a minimal set of jointly sufficient statistics if such were available. Theorem 8 tells us how to improve on an unbiased estimator by conditioning on sufficient statistics. For some estimation problems this unbiased estimator, obtained by conditioning on sufficient statistics, will be an UMVUE. To aid in identifying those estimation problems for which a derived estimator is an UMVUE, the concept of completeness of a family of densities is useful.

Definition 22 Complete family of densities Let Xl' ... , Xn denote a random sample from the density f(; 9) with parameter space 8, and let T = t(XI , ... , XII) be a statistic. The family of densities of T is defined to be complete if and only if 8 6[~(T)] 0 for all 9 e e implies that P6[~(T) = 0] = I for all 9 e 8, where ~(T) is a statistic. Also, the statistic T is said to be complete if and only if its family of densities is complete.

111/
Another way of stating that a statistic T is complete is the following: Tis complete if and only if the only unbiased estimator of 0 that is a function of T is the statistic that is identically 0 with probability 1.

EXAMPLE 32 Let Xl' ... , Xn be a random sample from the Bernoulli density. The statistic T = Xl - X 2 is not complete since 8 o[Xt - X 2 ] = 0 and Xl - X 2 is not 0 with probability I. Consider the statistic T =

II

Xi'

Let ~(T) be any statistic that is a function of T for which 8 8[~(T)] == 0

UNBIASED ESTIMATION

325

for all (J e E'l, that is, for 0 < (} < 1. To argue that Tis complete, we must show that A:(t) = 0 for t = 0, 1, ... , n. Now
8 9 [4T)] =
t=O

I A:(t)(n) 8'(1 t
=

6)n-t

= (l

- fJ)n

hence, 8 9 [A:(T)]

t-O

t ~(t)(~)

[0/(1 - O)J;

for all 0 < 6 < 1 implies that

or

for all a, where a = 81(1 - 8). Now in order for a polynomial in a to be identically 0, each coefficient of at, t = 0, ... , n, must be 0; that is,
x( t) (;)

= 0 for t = 0, ... , n, but (~) #

0; so x( t)

= 0 for t = 0, ... ,n. 11/1

EXAMPLE 33 Let Xh ... , Xn be a random sample from the uniform distribution over the intet:Val (0, 8), where E'l = {6: 0 > O}. Show that the statistic Yn is conlplete. We must show that if 8 9 [A:(Yn)] == 0 for all 8 > 0, then Po[A\Yn) = 0] == 1 for all 8 > O.
8 o[A\Yn)]
=

J4.y)frn(Y) dy = J 4y)0-nnyn-l dy,

(J

and 8o[A:(Yn)]

= 0 for all 0 > 0 when and only when

n
lln

r A:(y)yn-t dy = 0
"'0
(J

for all 0> 0

or

JA:(y)yn-l dy =0
o

for all 8 > O.

Differentiating both sides of this last identity with respect to 0 produces A:(8)on-l = 0, which in turn implies that A:(fJ) 0 for 8> o. //1/

In general, demonstrating completeness can require tricky analysis. The two above examples are exceptions. We state now, without proof, a theorem that gives us completeness of a statistic. It will be our main tool for arguing completeness.

326

pARAMETRIC POINT ESTIMATION

vn

Theorem 9 Let Xl' ... , Xn be a random sample fromf( ; 8), 8 E 8, where e is an interval (possibly infinite). If f(x; 6) = a(8)h(x) exp [c(B)d(x)], that is, f( .; 8) is a member of the one-parameter exponential family, IIII then L d(Xi ) is a complete minimal sufficient statistic.
Theorem 9 shows once again the importance of the exponential family or exponential class. We are finally adequately prepared to state the theorem that is useful in finding UMVUEs.

Theorem 10 Lehmann-Scheffe Let Xb ... , Xn be a random sample from a density f(; 8). If S = O'(Xb ... , Xn) is a complete sufficient statistic and if T* = t*(S), a function of S, is an unbiased estimator of -r(8), then T* is an UMVUE of -r(8).
Let T' be any unbiased estimator of -r(B) which is a function of S; that is, T' = t'(S). Then ~(I[T* - T'] == 0 for a1l 8 E 6, and T* - T' is a function of S; so by completeness of S, P(I[t*(S) = t'(S)] = 1 for all 8 E 6. Hence there is only one unbiased estimator of -r(8) that is a function of S. Now let T be any unbiased estimator of -r(8). T* must be equal to ~[T IS] since ~[T IS] is an unbiased estimator of -r(8) depending on S. By Theorem 8, var(l [T*] < var(l [T] for all 8 E 6; so T* is an UMVUE. IIII
PROOF

Let us review what this important theorem says: First, if a complete sufficient statistic S exists and if there is an unbiased estimator for -r(8), then there is an UMVUE for -r(8); second, the UMVUE is the unique unbiased estimator of -r(B) which is a function of S. To actually find that unbiased estimator of -r(B) which is a function of S, we have several ways of proceeding. First, simply guess the correct form of the function of S that defines the desired estimator. Second, guess or find any unbiased estimator of -r(8), and then calculate the conditional expectation of the unbiased estimator given the sufficient statistic. Third, solve for t*() in the equation 8(1[t*(S)] = -r(B). Such an equation becomes the integral equation t*(s)fs(s) ds = -r(8) in the case of a continuous random variable S and becomes the summation L t*(s)fs(s) == -r(8) for S a discrete random variable. We will employ two of these methods in the following examples.

EXAMPLE 34 Let Xl' ... , Xn be a random sample from the Poisson density

e-.ll x
f(x; l) =

x.

for x

0, 1, ....

UNBIASED ESTIMATION

327

We saw in Example 25 that/ex; 2) belongs to the exponential family with d(x) = x. By Theorem 9, the statistic L Xi is complete and sufficient. To :find the UMVUE of A itself, it suffices to guess a function of L Xi whose expectation is 2. Noting that 2 is the population mean, (lIn) L Xi is the obvious choice; so (lIn) L Xi is the UMVUE of A. Consider now estimating rCA) = e- A = P[Xi = 0]. (Recall Example 30.) Let us derive the UMVUE of e- A by calculating the conditional expectation of some unbiased estimator given the sufficient statistic. Any unbiased estimator will do as the preliminary estimator whose conditional expectation ~eeds to be calculated; so we may as well choose one that would make the calculations easy. I{o}( Xl) is an unbiased estimator of e- A and is relatively simple since it can assume only the two values 0 and 1. By Theorem 10, tf[I{o}(X1) IL Xi] is the UMVUE of e- A To find the desired conditional expectation, we first find the conditional distributi on of Xl given L Xi'

Therefore,
t9'[l{o)(X,)IL Xi =s]

= PIX, =OIL Xi =s] =

~ It

hence

h A t e UMVUE of e- for n > 1. For n = 1, I{o}(X1) is an unbiased estimator which is a function of the complete sufficient statistic X, and hence I(o}(X1) itselfis the UMVUE of e- A The reader may want to ~erive the mean and variance of
IS

328

PARAMETRIC POINT ESTIMATION

VII

and compare them with the mean and variance of the estimator
(lin)

II

I{o}(X,) given in Example 30.

i=l

IIII

EXAMPLE 35 Let Xl' ... , XII be a random sample from I(x; 0) = Oe- 8xI(o.oo)(X). Our object is to find the UMVUE of each of the following functions of the parameter 0: 0, 1/0, and e- K8 = P[X > K] for given K. Since Oe- 8xI(o. oolx) is a member of the exponential class (see Example 24), the statistic S = X n = (lIn)
II

Xi is complete and sufficient.

I Xi' i I

II

which is a function of the complete sufficient

statistic S

I Xi, is an unbiased estimator of 110; hence by Theorem 10, i= 1

elI Xi'
1

X II is the UMVUE of 110. To find the UMVUE of 0, one might suspect that the estimator is
n

of the form

where

e is a constant which may depend on n.

Now

for n > 1. So ~9[eII Xi] = 0 when e = n - 1; hence (n - l)/I Xi is the UMVUE of 0 for n> 1. The variance of (n - l)/I Xi is given by 02j(n - 2) for n > 2. Although one might be able to guess which function of S = I Xi is an unbiased estimator for e- K8 , let us derive the desired estimator by starting with the following simple unbiased estimator of e- K9 : I(K.oo)(X1), Note that C8[I(K,oolX1 )] = 0 . P[XI < K] + 1 . P[XI > K] = P[XI > K] = e- K9 ; so I(K, oo)(X1) is indeed an unbiased estimator of e- K8 , and therefore by Theorems 8 and 10 C 9[I(K, oo)(X1) IS] is the UMVUE of e- K9. Now, C8 [I(K , oo)(XI ) IS = s] = P[I(K. oo)(X1) = 11 S = s] = P[X1 > K IS = s). In order to obtain P[XI > K IS = s], we will first find the conditional distribu.. tion of Xl given S = s.

s
IX1IS .(Xll s) ~XI

UNBIASED ESTIMATION

329

IXbS<XI' s) Axl ~s lS<s) As

P[Xl <XI <X1

+ Ax l ; S

~~~------~~---.~~------~ (l/r(n)]8"s" Ie 6s ~s

<*X,

< S+

AS]

P[XI <XI <XI

+Ax1; S- X1 <tXI < S -XI +A5]

[1/r( n ) ]8"s" Ie
6s

~~--------~~~~~~~~----------~

~s

P[x I <

XI < XI + Ax I]P [s -

XI <

X, < S- XI +

AS]

[l/r(n)]8"s" Ie

68

~s

for x I < s and n > L

cS'6[I(K,oo)(X 1 )!S=S]=P[X l >K!S=s]= fKoofxdS=.,cXlls)dX1

_ JS
-

r(n) (s - Xt)"-2 Kr(n-l) S,,-1 dXt

n -1

,,-1

fO
s-K

y"-2( -dy)
s-K
0

n-

1 y,,-t
n - 1

S,,-l

for s > K and n > 1, where the substitution y

=s-

Xl

was made.

Hence,

330

pARAMETRIC POINT ESTIMATION

VII

is the UMVUE for e-K(J for n > 1. (Actually the estimator is applicable for n = 1 as well.) It may be of interest and would serve as a check to verify directly that

is unbiased.

where the substitution u = s - K was made.

//1/

In closing this section on unbiased estimation, we make several remarks. Remark F or some functions of the parameter there is no unbiased estimator. For example, in a sample of size 1 from a binomial density there is no unbiased estimator for 1/8. Suppose there were; let T = t(X) denote it.

= 1/8, which says that an nth-degree polynomial in 8 is identical to 1/8, which cannot be. ///1
Then 8(J[T] =
t(x)

x=O

(n) 9"(1 x

8)n-x

Remark We mentioned in Subsec. 5.1 that the Cramer-Rao lower bound is not necessarily the best lower bound. For example, the Cramer-Rao lower bound for the variance of unbiased estimators of 8 in sampling from the negative exponential distribution is given by 82 /n (see Example 29), and the variance of the UMVUE of 8 is given by 82 /(n - 2) (see Ex//11 ample 35). 82 /(n - 2) is necessarily the best lower bound. Remark For some estimation problems there is an unbiased estimator 1/1/ but no UMVUE. Consider the following example.

EXAMPLE 36 Let Xb ... , Xn be a random sample from the uniform density over the interval (8, 8 + 1]. We want to estimate 8. X n -! and (Y1 + Yn)/2 -! are unbiased estimators of 8, yet there is no UMVUE

LOCATION OR SCALE JNVARlANCE

331

of (J. For fixed 0 <p < 1, consider the estimator g(XI - p) + p, where the function g(y) is defined to be the greatest integer less than y. Now
8[g(Xl- p) + p]

=1
For fixed
(J

9+1

g(Xl - p) dXl

+p =

6+1-p

g(y) dy

+ p.
satisfying

6-p

and p, there exists an integer, say N () - p < N < (J + 1 - p. Hence,

S[g(Xl - p)

= N(B, p),

p]

8+ 1- p

g(y) dy

+p=

6-p

(N - 1) d y

f6 + I - p
N

N dy

+p=

(}.

6-p

So g(XI - p) + p is an unbiased estimator of B. Moreover, if (} + 1 - P is an integer, say J, then g(XI - p) = J - 1 for all Xl satisfying (} - p < Xl - P < B + 1 - p; so g(Xl - p) + p = J - 1 + p = B + 1 - P - 1 + p = B for all B < Xl < B + 1; that is, g(XI - p) + p estimates B with no error, and hence has zero variance for B + 1 - P equal to any integer. So, we have an estimator, namely g(Xl - p) + p, which has zero variance for (} = any integer - 1 + p. But 0 < P < 1 is arbitrary; so for any fixed B, say Bo , we can find an unbiased estiumtor of B which has zero variance at (}o' Hence, in order for an estimator to be the UMVUE of B, it must have zero variance for all (}; that is, it must always estinlate B without error. Qearly, no such estimator exists. {The reader may wish to show that var [g(Xl - p) + p] = [N - (B - p))[(B + 1 - p) - N], where N = N(}, p) is an integer satisfying B - p < N B + 1 - p.} //// Remark It is sometimes possible to find an UMVUE even when a minimal sufficient statistic is not complete. See Prob. 11, p. 313, in Rao [17]. fIll

LOCATION OR SCALE INVARIANCE

In the last section we employed the property of unbiasedness as a means of restricting the class of estimators with the hope of finding an estimator having minimum mean-squared error within the restricted class. In this section we will indicate how an alternative property, the property of invariance, can be used to restrict the class of estimators. Our discussion will be limited to only two types of invariance, namely, location invariance and scale invariance; a fuller discussion, which is beyond the scope of this book, can be found in Refs. [12] and [19].

332

PARAMETRIC POINT ESTIMATION

VII

6.1

Location Invariance

If the observations Xl' .. , Xn represented measurements of some sort and the parameter being estimated was also measured in the same units, one might reasonably require that an estimator t(, ... , .) satisfy the property t(XI + c, X2 + c, "" xn + c) = t(XI' "', xn) + c for every constant c. The idea is that if a constant c is added to each of the measurements Xl' "', Xn' then the estimator evaluated at the adjusted measurements Xl + c, ... , Xn + c ought to adjust the estimated values t(XI' .. , xn) by adding the same constant to it. For example, suppose that it is desired to estimate the average weight of a group of pigs when the only method available for weighing is for a person to stand on a scale holding a pig; so both the pig and person are weighed. If one person were to hold the pigs, the measurements (weights) Xl + c, ... , Xn + c would be obtained, where Xi is the weight of the ith pig and c is the person's weight. If, on the other hand, someone else were to hold the pigs, the measure~ ments Xl + c', ... , xn + c' would be obtained, where c' is the other person's weight. It seems reasonable that the estimate of the average weight of the group of pigs obtained should not depend on which person held the pigs; that is, the estimate should not vary with c, the weight of the pig holder. We define a location-invariant estimator accordingly.

Definition 23 Location invariant An estimator T = t( Xl' ... , Xn) is defined to be location-invariant if and only if t(XI + c, ... , xn + c) = t(Xb "', xn) + c for all values Xb .. , xn and all c. IIII
A number of the estimators that we have encountered are locationinvariant, for example, X nand (Yi + Yn)/2, as the following shows:
t(XI

+ c, "

., Xn + c) =

L (Xi

+ c)

= --

LXi

+ c = t(XI' ... , xn) + c

for t(XI' ... , xn) = xn; and

t(XI

+ c, ... , Xn + c)
min [Xl

+ c, ... , Xn + c] + max
2

[Xl

+ c, ... , Xn + c]
Xn]

min [Xl' ... , Xn] + c + max [X I, 2

... ,

+C

LOCATION OR SCALE INVARlANC

333

for t(X., . , XII) = (YI + y,J/2. On the other hand, quite a number of 2 estimators are not location-invariant; for example, 8 and YII - Y1, as the fol2 2 lowing shows: Take T = t(XI , ... , XII) = 8 = L (Xi - XJ /(n - 1); then t(XI + C, , XII + c) = L [Xi + c - L (Xi + c)ln]2/(n - 1) = t(X., ... , XII)' instead of t(XI' . , xJ + c. Now take T = t(XI' ... , X,J = Y,. - Yi; then t(XI + c, .. , X,. + c) = max [Xl + c, ... , X,. + c] - min [Xl + c, ... , Xn + c] = max [Xl' ... , xn] + c - min [Xl' ... , X,.] - C = t(X., ... , x,J, instead of t(XI' .. , XII) + c. Our use of location invariance will be similar to our use of unbiasedness. We will restrict ourselves to looking at loc~tion-invariant estimators and seek an estimator within the class of location-invariant estimators that has uniformly sma1lest mean-squared error. The property of location invariance is intuitively appealing and turns out also to be practically appealing if the parameter we are estimating represents location. be a family of densities indexed by a parameter 8, where 9 is the real line. The parameter e is defined to be a location parameter if and only if the density f(x; 8) can be written as a function of X - e; that is, f(x; 8) = h(x - 8) for some function h( . ). Equivalently, e is a location parameter for the density fx(X; e) of a random variable X if and only if the distribution of X - e does not depend on e. 111/ We note that if e is a location parameter for the family of densities {f( . ; 8), e E 8}, then the function h( . ) of the definition is a density function given by h( . ) = f( ; 0).

Definition 24 Location parameter Let {f( ;

e), e E 9}

EXAMPLE 37 We will give examples of several different location parameters. Iff(x; e) = <P(J. I (X), then e is a location parameter since
1 tf> ,(x) = J2tt exp -

[12(x - 8)2] = <Po. ,(x - 8).

Or if X is distributed normally with mean e and variance I, then X - 8 has a standard normal distribution; hence the distribution of X - 8 is independent of e. If f(x; e) = 1(6 - i. 6 + t lx), then e is a location parameter since f(x; (J) = 1(6 -to 9 +t)(X) = 1( -t. t)(X - e), a function of X - e. If f(x; a) = 1/11:[1 + (x - a)2], then a is a location parameter since f(x; a) is a function of X-a. IIII
A

334

PARAMEI'RlC POINT ESTIMATION

vn

We will now state, without proof, a theorem that gives within the class of location-invariant estimators the uniformly smallest mean-squared error estimator of a location parameter. The theorem is from Pitman [41].

Theorem 11 Let X., ... , Xn denote a random sample from the density I( . ; e), where e is a location parameter and is the real line. The estimator

t(X 1 , ..

f etl !(X ,X )=
n
n

t;

e) de
(17)

f}]l!(X i ; e) de
is the estimator of e which has uniformly smallest mean-squared error //// within the class of location-invariant estimators.

Definition 25 Pitman estimator for location The estimator given in Eq. (17) is defined to be the Pitman estimator for location. IIII
According to the formula given in Eq. (17), determining the Pitman estimator requires evaluating the integrals given in the numerator and denominator; such evaluation may not be easy. Note that the integration is with respect to the parameter; SO the resulting ratio will be a function of Xl, ... , X n.

EXAMPLE 38 Let Xl' ... , Xn be a random sample from a normal distribution with mean e and variance unity. We saw in Example 37 that e is a location parameter. Our object is to find the Pitman estimator of e, which is given by Eq. (17). In the following series of equalities one should be forewarned that cancellations and insertions are being made simultaneously in the numerator and denominator.

Je(I/J~)n exp [-t I

(Xi -

e)2] de
8)2] de

f(I/J2n)n exp [-t L (Xi -

-----------------------

LOCATION OR SCALE INV ARlANCE

335

I8 exp .[ -(nI2)(8 - XIIi] d8 I exp [- (nI2)(8 - X ,,)2] d8

I 8[1/-J'~(1I-J';;)]exp{ -t[(8 - X n)/{lj-J';;)]2}dO I [l/J2n(lJJn)] exp{ -t[(8 Xn)/(I/-J';;)f} d8

=X n

by noting that the last denominator is just the integral of a normal density with mean Xn and variance lin and hence is unity, and the last numerator is the mean of this same normal density and hence is X". We note that, for this example, the Pitman estimator of 8, which is uniformly minimum mean-squared error among location-invariant estimators, is identical to the UMVUE of 8; that is, the estimator that is best among location-invariant estimators is also best among unbiased estimators. IIII

EXAMPLE 39 Let Xb ... , X" be a random sample from a uniform distribution over the interval (8 - t, 8 + t). According to Example 37, 8 is a location parameter. The Pitman estimator of 8 is

" f8 n I(O-t. 8+t)(X

i=1

i)

d8

I8.n l(x,-t. Xi + t)(8) d8

"

- ----------

p=1

i=1

Dl(8-t,0+tlXi)d8

JfI
i=1

I(X,-t,Xt+tl 8 )d8
Y1
=

J~!~t8 d8 _1 . (Y1 + t)2 - (Y" - t)2 J~!~t d8 - 2 (Y1 + t) - (Y" - t)

+ Y"
2

Recall that for this example there is no UMVUE of 8.

(See Example 36.)

Jill
Remark A Pitman estimator for location is a function of sufficient statistics.
PROOF

If 8 1

0l(Xh

. ,

Xn), . , 8.,.

0k(X1 ,

. ,

Xn) is a set of

sufficient statistics, then by the factorization criterion

I1 !(Xl;
i= 1

8) =

g(Sb , SJi;; 8)h(Xb . , x,,); so the Pitman estimator can be written as

336

PARAMETRIC POINT ESTIMATION

VII

f 8J} I(Xi;8) d8 f 8g(Sb ... , Sk; fJ)h(Xl' ... , Xn) d8 I in I(X 8) d8 fg(Sl' . '" Sk; 8)h(Xl' ... , Xn) d8 f 8 g(S S k; 8) d8 Ig(Sl' ... , Sk; 8) d8
n

i;

l' ... ,

which is a function of Sl' ... , Sk' 6.2

IIII

Scale Invariance

For those experiments in which measurements can be made in different units, such as length being measured in either inches or centimeters, weight being measured in either pounds or kilograms, or volume being measured in either quarts or liters, one might reasonably require that his statistical procedure be independent of the measurement units employed. If the statistical procedure is that of point estimation, then one might require that the estimator that is to be used satisfy the property of scale invarianee defined below. The idea is that an estimator will be scale-invariant if the estimator does not depend on the sca1e of the measurement.

Definition 26 Scale-invariant An estimator T = t(Xl' ... , XJ is defined to be scale-invariant if and only if t(exl' " . , eXn) = et(xl' .. , , xn) for all values Xl' ... , xn and all e > O. IIII
A number of the estimators that we have considered are scale-invariant, including X n , JS2, (Yl + Yn)/2, and Yn - Y Our discussion of scalel . invariant estimators will be limited to problems concerning estimation of scale parameters defined below.

Definition 27 Scale parameter Let {f( . ; 8), 8 > O} be a family of densities indexed by a real parameter 8. The parameter 8 is defined to be a scale parameter if and only if the density I(x; 8) can be written as (1/8)h(xI8) for some density h(). Equivalently, 8 is a scale parameter for the density Ix(x; 8) of a random variable X if and only if the distribution of XI8 is independent of 8. / / //
Note that if 8 is a scale parameter for the family of densities {/(' ; 8),8 > O}, then the density h( . ) of the definition is given by hex) = I(x; 1).

LOCATION OR SCALE INVARIANCE

337

EXAMPLE 40 We give several examples of scale parameters. If f(x; 1) = (l/l)e-x/ Al 0, t;1)ix ), then 1 is a scale parameter since e~Y1(0, oo)(Y) is a density. ~ote that this parameterization of the negative exponential distribution is not the parameterization that we have used previously. If
f(x; 6)

"'0, .,(x) = J;1tlT exp [ - H~)

then

is a scale parameter since (I/J2ic) exp (-ty2) is a density. If f(x; 0) = (1/ 8)1(0. 9) (x) = (l/8)I(o.1)(xl8), then 0 is a scale parameter since I(O.l)(Y) is a density. If f(x; 8) = (lIO)I(9. 29)(X) = (1/8)1(1, 2lx /8), then 8 is a scale paramII/I eter since 1(1. 2)(Y) is a density.
(J

Our sole result for scale invariance, a result that is comparable to the result of Theorem lIon location invariance, requires a slightly different frame work. Instead of measuring error with squared-error loss function we measure it with the loss function t(t; 8) = (t - 6)216 2 = (tiO - 1)2. If It - OJ represents error, then 1001 t - 01/6 can be thought of as percent error, and then (t - 8)2/62 is proportional to percent error squared. We state the following theorem, also from Pitman [41], without proof.

Theorem 12 Let X b . , Xn bearandomsamplefromthedensityf(' ;8), where 0> 0 is a scale parameter. Assume that f(x; 0) = 0 for x < 0; that is, the random variables Xi assume only positive values. Within the class of scale-invariant estimators, the estimator

has uniformly smallest risk for the loss function t(t; 0) = (t - 8)2/02. ...

/1//

Definition 28 PItman estimator for scale The estimator given in Eq. (18) is defined to be the Pitman estimator for scale. /111 Remark The Pitman estimator for scale is a function of sufficient IIII statistics.

338

PAR.AMEI'RlC POINT ESTIMATION

VII

EXAMPLE 41 Let Xl' ... , Xn be a random sample from a density I(x; lJ) (1/0)/(0,6)(x). The Pitman estimator for the scale parameter B is

frO (110 2) fI (1/0)1(0, 8)(X ,) dO

o
1=1

f o-n00

00 2 _ Y..:. :;.."_ __

dO

3 fo (1/0 )

00

lU1(1/0)/(0. 6)(X,) dO

f " o-n- 3 dO
y

{1/[(n = {1/[(n

+ 3) - 1]}yn -<n+3)+ 1 =

+ 2) _1]}yn -(n+2)+1

n n

+2
+ 1 Yn'

We know that Yn is a complete sufficient statistic and 8[Yn ] =[nl(n + 1)]0; so by the Lehmann-Scheffe theorem [en + 1)/n]Yn is the UMVUE of o. IIII

EXAMPLE 42 Let X., ... , Xn be a random sample from the density f(x; A) = (IIA) exp (-xIA)/(o. oolx). The Pitman estimator for the scale parameter A is

n 2 fo (1/A )IIJ !(X , ; A) dA 1

00

t
-

00

(1IAn+2)exp( -

I: XdA)dA

(I/A

D,f(X,; A)dA

('(I/An + 3)eXp ( -

L XJA)dA

t (<x/I:
00

x,)n+2 e-O:(I: X,/<x 2) d<x

00

(<x/I: Xi) n+3 e -(I: X,J<x2) d<x

= I: x. r(n + 1) = I: X,
'r(n + 2)
n

+1

(It can be shown that the UMVUE of A is Xi/n.) Note that X,Jn is a scale-invariant estimator, and, hence, since Xil(n + 1) is the sca1e-invariant estimator having uniformly smallest risk for the loss function (t - Of 102 , the risk of Xi/en + 1) is uniformly 2 smaller than the risk of Xi/n. Also, since here risk equals 1/0 times the Xil(n + 1) is uniformly smaller than the MSE of MSE, the MSE of Xil n . 1/1/

I:

I:

I:

I:

I:

I:

I:

BAYES ESTIMATORS

339

7 BAYES ~STIMATORS
In our considerations of point-estimation problems in the previous sections .of this chapter, we have assumed that our random sample came from some densIty f( . ; 8), where the function/( . ; .) was assumed known. Moreover, we have assumed that 8 was some fixed, though unknown to us, point. In some realworld situations which the density f( . ; 8) represents, there is otten additional infonnation about 6 (the only assumption which we heretofore have made about 8 is that it can take on values in 8). For example, the experimenter may have evidence that 6 itself acts as a random variable for which he may be able to postulate a realistic density function. For instance, suppose that a machine which stamps out parts for automobiles is to be examined to see what fraction o of defectives is being made. On a certain day, 10 pieces of the machine's output are examined, with the observations denoted by Xl' X 2 , , X 10 , where Xi = 1 if the ith piece is defective and Xl = 0 if it is nondefective. These can be viewed as a random sample of size 10 from the Bernoulli density for 0 <0< 1, which indicates that the probability that a given part is defective is equal to the unknown number O. The joint density of the 10 random variables Xl' X 2 , , XlO is for 0 :::; 0 < 1. The maximum-likelihood estimator of 0, as explained in previous sections, is = X. The method of moments gives the same estimator. Suppose, however, that the experimenter has some additional information about 8; suppose that he has observed that on various days the value of 0 changes and it appears that the change can be represented as a random variable with the density

ge(O) = 60(1 - 0)/[0,1](0).

An important question is: How can this additional information about 0 be used to estimate 00 , where 60 is the value that was equal to on the day the sample was drawn? To examine this problem, we will assume, in addition to the assumption that our random sample came from a density 1(' ; 8), that the unknown parameter 8 is the value of some random variable, say . We wUI still be interested in estimating some function of 0, say 1'(0). If is a random variable, it has a distribution. We let G( .) = Ge (') denote the cumulative distribution function of and g(.) = ge(') denote the density function of and we assume these . ' functIons contaIn no unknown parameters. In order to emphasize that the

340

PARAMETRIC POINT ESTIMATION

VII

distribution of E> is over the parameter space, we have departed from our custom of using F( .) and f( .) to represent a cumulative distribution function and density function, respectively, and have used G( . ) and g( . ) instead. If we assume that the distribution of E> is known, we have additional information. So an important question is: How can this additional information be used in estimation? It is this question that we will address ourselves to in the foll<?wing two subsections. In many problems it may be unrealistic to assume that B is the value of a random variable; in other problems, even though it seems reasonable to assume that B is the value of a random variable E> the distribution of E> may not be known, or even ifit is known, it may contain other unknown parameters. However, in some problems the assumption that the distribution of E> is known is realistic, and we shall examine this situation.

7.1

Posterior Distrib ution

Heretofore we have used the notationf(x; B) to indicate the density of a random variable X for each B in 8. Whenever we want to indicate that the parameter B is the value of a random variable E>, we shall write the density of X as f(x 1 B) instead of f(x; 8). We should note that f(x 1 B) is a conditional density; it is the density of X given E> = B. A more complete notation for f(x 18) would be fXIS=O(xl B). Let X h . . . , Xn be a random sample of size n from the density f( 18), where B is the value of a random variable E>. Assume that the density of E>, gs(), is known and contains no unknown parameters, and suppose that we want to estimate L(B). How do we incorporate the additional information of known gs( . ) into our estimation procedures? In the past, we thought of the likelihood function as a single expression that contained all our information; the likelihood function included the observed sample Xh .. , Xn as well as the form of the density f(x; B) we sampled from in its expression. Now we need an expression that contains all the information that the likelihood function contains plus the added information of the known density gs()' gs() is called the prior distribution of E>. It summarizes what we kn.ow about B prior to taking a random sample. What we seek is an expression that summarizes what we know about B after we take a random sample. We seek the posterior distribution of E> given Xl = X h ... , Xn = x n

Definition 29 Prior and posterior distributions The density gs(') is called the prior distribution of E>. The conditional density of E> given Xl = Xl' . , Xn = x n , denoted by fSlxl =Xl .....Xn =xn(B 1 Xl' , xJ, is called the posterior distribution of E>. IIII

BAYES ESTIMATORS

341

Remark

= for random sampling. fXIY=:V<x I y)fy(y)Lfx(x).]

f La I(X,16)]g,,(8) d8
=

[,D, I(x" 6)]U,,(6)

(19)

[Recall that fYlx=iylx) =fx,y(x, y)lfx<.x)

1///

The posterior distribution replaces the likelihood function as an expression that incorporates all information. If we want to estimate 6 and parallel the development of the maximum-likelihood estimator of 6, we could take as our estinlator of 6 that 6 which maximizes the posterior distribution, that is, estimate 6 with the mode of the posterior distribution. However, unlike the likelihood function (as a function of 6), the posterior distribution is a distribution function; so we could just as well estimate 6 with the median or mean of the posterior distribution. We will use the mean of the posterior distribution as our estimate of 6, and in general we could estimate 1'( 6) as the mean of 1'( 9) given Xl = Xl, , X" = x,,; that is, take 8[1'(9) I Xl = Xh ... , X" = x,,] as our 'estimate of 1'(6).

Definition 30 Posterior Bayes estimator Let Xh .. , X" be a random sample from a density f(x 16), where 6 is a value of the random variable 9 with known density ge(). The posterior Bayes estimator of 1'(6) with respect to the prior ge(') is defined to be
8[ 1'(9) I X h
... ,

X,,].

(20)

1///
Remark
8[1'(9)1 Xl =
Xl' '

X"

= X,,] = J1'(6)!elxl=Xl .... Xn:;x.J6Ixh ... ' x,,) d6

f r(8>ta I (x, 8 )] g,,(8) d8 f La I(X.l8)] g.,(8) d8 .

1

(21)

/111

One might note the Similarity between the posterior Bayes estimator of 1'(6) = 6 and the Pitman estimator of a location parameter [see Eq. (17)].

342

PARAMETRIC POINT ESTIMATION

vn

EXAMPLE 43 Let Xl' ... , Xn denote a random sample from the Bernoulli density f(x 18) = F(1 - e)l-x for x = 0, 1. Assume that the prior distribution of 9 is given by gs(8) = /(0,1) (8); that is, 9 is uniformly distributed over the interval (0, 1). Consider estimating 8 and T(e) = 8( 1 - 8). Now

so the posterior Bayes estimator of 8 with respect to the uniform prior distribution is given by

8[9 IXl

= Xh

.. ,

Xn =

xJ
+ 2, n - LXi + 1) + 1, n - LXi + 1)

f 8!SIXl=Xlo ...,Xn=xn(Olxh'' Xn) d8

- U

_ J~88rX'(1 - 8)n-rXi d8 _ B(L Xi 8r XI( 1 - 8t - r Xi d8 - B(L Xi

_ r(L Xi + 2)r(n - LXi + 1) . r(n + 2) r(n + 3) r(L Xi + l)r(n - L Xi + 1)

LXi + 1
n

+2 .

Hence the posterior Bayes estimator of 8 with respect to the uniform prior distribution is given by (L Xi + 1)/(n + 2). Contrast this to the maximum-likelihood estimator of 8, which is Xt!n. Xt!n is unbiased and an UMVUE, whereas the posterior Bayes estimator is not unbiased. To obtain the posterior Bayes estimator of, say T(8) = 8(1 - 8), we calculate

f 8(1- 8)!slx =xI, ...,xn=xn(8I x

1

l,"

xn) d8

U8(1 - 8)8rXi (1 - 8t- rxi d8 J~8r XI(1 - 8t- r Xi d8

=

r(L Xi + 2)r(n - L Xi + 2) r(n + 2) r(n+4) 'r(L x i+ 1)r(n-L x i+

(L Xi + l)(n - LXi + 1)
(n

1)

+ 3)(n + 2)

So the posterior Bayes estimator of 8(1 - 8) with respect to a uniform prior distribution is (L Xi + 1)(n - L Xi + l)/(n + 3)(n + 2). IIII

BAYES ESTIMATORS

343

We noted in the above example that the posterior Bayes estimator that we obtained was not unbiased. The following remark states that in general a posterior Bayes estimator is not unbiased.

Remark Let T~ = t~(XI' .,., XJ denote the posterior Bayes estimator of 't(6) with respect to a prior distribution G( ,). If both T~ and 1'(9) have finite variance, then either var [T~ 18] = 0, or T~ is not an unbiased estimator of 't(6). That is, either T~ estimates 1'(8) correctly with probability 1, or T~ is not an unbiased estimator of 1'(8),
Let us suppose that T~ is an unbiased estimator of 1'(8); that is, 8[T~ 18] = 1'(8), By definition we have T~ = t~( X., .. , Xn) = 8[1'(9)1 X., .. , Xn]. Now
PROOF

var [T~] = 8[var [T~I 9]]

=

+ var [8[T~1 9]] 8[var [T~I 9]] + var [1'(9)],

Xn]]

and var ['t(9)]

= 8[var [1'(9)1 X., . , Xn]] + var [8[1'(9)1 Xl"'"

= 8[var [1'(9) 1X., .. ,
Xn]]

+ var [T~];

hence, 8[var [T~I 9]] + 8[var [1'(9)1 Xl"'" Xn]] = O. Since both 8[var [T~ 19]] and 8[var [1'(9) 1X., . " Xn]] are nonnegative and their 19]] = 0, and since sum is 0, both are O. In particular, 8[var var [T~I 9] is non-negative and has zero expectation, var [T~18] = O. Jill

[n

7.2 Loss-function Approach

In Subsec. 3.4 we introduced the concepts of loss and risk. These two concepts were used to assess goodness of estimators. In this section we discuss how the additional information of knowledge of a prior distribution of 9 can be used in conjunction with loss and risk to define or select an optimum estimator. We commence with a review of the problem we hope to solve. Let X., ... , Xn be a random sample from a density f(xl8), 8 belonging to e, where the function f( IfJ) is assumed known except for 8. We assume that the unknown 8 is the value of some random variable 9 and that the distribution of 9 is known and contains no unknown parameters. On the basis of the random sample X., .. , Xn we hope to estimate 1'(8), some function of 8. In addition, we assume that a loss function I(t; 8) has been specified, where I(t; 8) represents the loss incurred if we estimate 1'(8) to be t when 8 is the parameter of the density from which we sampled, For any estimator T = t( X., ... , X J, we

344

pA.RAMBTlUC POINT ESTIMATION

vn

noted in Subsec. 3.4 that Co[t(T; B)] represented the average loss of that estimator, and we defined this average loss to be the risk, denoted by {!ttCB), of the estimator t(, ... , '). We further noted that two estimators, say 11 = tl(XI , ... , Xn) and Tz = t 2 (XH , X n), could be compared by looking at their respective risks {!t'l(B) and {!t,lB) , preference being given to that estimator with smaller risk. In general, the risk functions as functions of B of two estimators may cross, one risk function being smaller for some B and the other smaller for other B. Then, since B is unknown, it is difficult to make a choice between the two estimators. The difficulty is caused by the dependence of the risk function on B. Now, since we have assumed that Bis the value of some random variable 9, the distribution of which is also assumed known, we have a natural way of removing the dependence of the risk function on B, namely, by averaging out the B, using the density of 9 as our weight function.

Definition 31 Bayes risk Let Xl, ... , Xn be a random sample from a

density f(x IB), where B is the val ue of a random variable 9 with cumulative distribution function G(') = Ge ( .) and corresponding density g( . ) = ge( '). In estimating ,(0), let t(t; B) be the loss function. The risk of estimator T = t(Xt , , XII) is denoted by {!tiB). The Bayes risk of estimator T = t(XH ... , Xn) with respect to the loss function t( .; .) and prior cumulative distribution G(')' denoted by ~(t) = ~(., G( I), is defined to be
-l( I)

= -l/, G( I) = ~ {!tiB)g(B) dB.

(22)

fIll
The Bayes risk of an estimator is an average risk, the averaging being over the parameter space ~ with respect to the prior density g(.). For given loss function t( .; .) and prior density g( . ) the Bayes risk of an estimator is a real number; so now two competing estimators can be readily compared by comparing their respective Bayes risks, still preferring that estimator with smaller Bayes risk. In fact, we can now define the best" estimator of ,(B) to be that estimator with smallest Bayes risk.
U

Definition 32 Bayes estimator The Bayes estimator of ,(B), denoted by T,*, G = ti, G(X1 , . , XJ, with respect to the loss function t( .; .) and
prior cumulative distribution G('), is defined to be that estimator with smallest Bayes risk. Or the Bayes estimator of ,(B) is that estimator t1. G satisfying -l/, G( t*) = ~/, G( ti, G) .s;; -l/, G( I) for every other estimator T = t(XI' ... , X,.) of ,(B). 11II

BAYES ESTIMATORS

345

The posterior Bayes estimator of T(8), defined in Definition 30, was defined without regard to a loss function, whereas the definition given above requires specification of a loss function. The definition leaves the problem of actually finding the Bayes estimator, which may not be easy for an arbitrary loss function, unsolved, However, for squared-error loss, finding the Bayes estimator is relatively easy, We seek that estimator, say 1*( , "', .), which minimizes the expression J~ BitCe)g(e) de = Iii C8 [[/(Xb ... , x,.) - 1:(e)]2]g(e) de as a function over possible estimators I( , . " .). Now,

J~ Co[[/(X 1,
=

, ,

Xn) - T(e)]2]g(e) de
T(6)]2IxI, ... xJX., .. , xnl 8)

JiilJ~

r f r [/(XH ... , Xn) -

f~(i [0) t~
IXI X ..(Xl'

=1

Ii dXi}g(e) de

I(x" ... , x,,) l' Ix .. .... x.(x" ... , x.1 O)g(O) dO}

Ix

10

x..(X h

.. ,

Xn)

, Xn)

1= 1

n dx, n
n

IXl, X ..(x.,

... ,Xn)

dXi'

i= 1

and since the integrand is nonnegative, the double integral can be minimized if the expression within the braces is minimized for each x., ... , X n But the expression within the braces is the conditional expectation of [T(e) - I(XH ... , xn)]2 with respect to the posterior distribution of e given Xl = Xl' ... t Xn = X n , which is minimized as a function of I(x., ... , xn) for I*(x., ... , xn) equal to the conditional expectation of T(e) with respect to the posterior distribution of e given Xl = Xl' .. , Xn = x n {Recall that C[(Z - a)2] is minimized as a function of a for a*"= C[Z].} Hence the Bayes estimator of T(e) with respect to the squared-error loss function is given by

(23)

which is identical to the estimator given in Eq. (21).

346

pARAMETRIC POINT ESTIMATION

vu

For a general loss function, we seek that estimator which minimizes J ~ 9flO) g(O) dB. Again,

f- 9fJ.B)g(B) dB =fft [J/(t(x" ... , x,); IJ)fXh ... xJx" ... , x, I IJ) V, dx,]O(IJ) dIJ
~

J. [J/(

I(x" ... , x,); IJ)felx, =x" .... x.=x.(IJI x" ... , x,) dIJ]

fXl .. Xn(Xl' . ,

xn)

i= 1

n dx

i,

and minimizing the double integral is equivalent to minimizing the expression within the brackets, which is sometimes called the Rosterior risk. So, in general, the Bayes estimator of r(B) with respect to the loss function t( .; .) and prior density g( ) is that estimator which minimizes the posterior risk, which is the expected loss with respect to the posterior distribution of 9 given the observations Xl, ... , x n We have the following theorem and corollaries. Theorem 13 Let Xl, .. , Xn be a random sample from the density f(x IB), and let g(B) be the density of 9. Further let t(t; B) be the loss function for estimating r(B). The Bayes estimator of r(O) is that estimator 1*( . , ... , .) which minimizes

as a function of

Ie , ... , .).

IIII

Corollary Under the' assumptions of Theorem 13, the Bayes estimator of r(B) is given by

(25)

for a squared-error loss function.

IIII

Corollary Under the assumptions of Theorem 13, the Bayes estimator of B is given by the median of the posterior distribution of 9 for a loss IIII function equal to the absolute deviation. The proofs of the theorem and first corollary preceded the statement of the theorem. The second corollary follows from the observation that

BAYES ESTIMATORS

347

f._I (J ~

t(Xh""

x,,)lfs1xl=xl ..... XI'l=xl'l(BI Xl

, , X,,)

dB

is minimized as a function of I( . , ... , .) for 1*( . , ... , .) equal to the median of the posterior distribution of 8. {Recall that &[ IZ - a I] is minimized as a function of a for a* = nledian of Z.} EXAMPLE 44 Let Xl, ... , X" be a random sample from the normal density with mean B and variance 1. Consider estimating B with a squared-error loss function. Assume that 8 has a normal density with mean 110 and variance 1. Write 110 = Xo when convenient. According to Eq. (25) the Bayes estimator is given as the mean of the posterior distribution of 8.
fSlx 1 =X1. Xn=xl'l(B IXb
. ,

X,,)
tV/(X;!

= Ix" ... x.llx!> ... , x.IO)g(O) _

Ix, ..... X.(x .. ... , x.)

0)] g(O)
t(0 - /lO)2]
t(O /lo)2] dO

[.V/(X,IO)]g(O) dO

--=--------~~~------~---------------------

r.,

(11

J21t)" exp [ - t

(x, - 0)2] (1/J21t) exp [ -

(1/.fiic) exp [ - t

(X, - 6)' ] (1/.fiic) exp [ -

exp [
exp

-t,t

{-t[(n + 1)02 - 21!,to x, + It/f)} f., exp {-t[(n + 1)62 - 20,t/, + txl]} dO
exp exp ( -[en

f., [-t,t.

(X,- 0)2]
(x, - 0)2] dO

+ 1)/2l{02 ~ 20

roo
_

exp

(-[(n + 1)/21{02 - 26

t t

xrl(n + 1) +

[t xr/(n + !)n) xrt(n + I) + [t xl/(n + on)

dO

[lIJ2n/(n + 1)] exp {-[(n + 1)/2l[0 -

xr/(n + l)rJ
dO

roo [IIJ2n!(n + 1)] exp (-[(n + 1)/2][0 - L xt!(n + 1)] 2}

=

1 e J2n/(n +1) xp

(_n+l[" ]21 2 ~xd(n + 1) J;

(J -

348

PARAMETRIC POINT ESTIMATION

VII

the denominator is unity since it is the integral of a density. We have shown that the posterior distribution of e is normal with mean L~ xi/(n + 1) and variance 1/(n + 1); hence the Bayes estimator of 8 with respect to squared-error loss is

Xo + L Xi Jlo + L Xi 1 1 --.....-.- --n+1 n+1

Since the posterior,distribution of the same; hence

ft

is normal, its mean and median are

n

Jlo+LXi
1

n+1

is also the Bayes estimator with respect to a loss function equal to the absolute deviation. II/I

EXAMPLE 45

Let Xl' ... , X" be a random sample from the density I(x 18) = (l/fJ)/(0.8)(x), Estimate 8 with the loss function let; fJ) = (t - 8)2/82. Assume that e has a density given by g(8) = 1(0.1)(8). Let y" denote max [Xl, ... ,xft ]. Find the posterior distribution of 9.
(l/fJ)ft

f8I X l=xlo Xn=Xn

( ll

t V Xl"'"

) X" =

n I(0.8)(x;)I(0. 1)(8)
ft

-l~--"-----

i= I

fo (1/8)"Xl I(0.8/X i) d8

(1/ 8 )"I(Yn.l)(8)

fYn(1/8)"I(Yn.n(8) d8
I

(1 18)nI (Yn. 1)(fJ)

= [l/(n - l)](l/y:-l - 1)'

We seek that estimator which minimizes Eq. (24), or we seek that estimator 1(') which minimizes

J{[/(Yn) -

8fI82}(1/~)/(Yn.1)(8) d8 [l/(n - l)](lly: 1 - 1)

BAYES ESTIMATORS

349

or that estimator which minimizes

[t(yJ - 0]2 (1'+ 2 dO

1n

= [t(y,.)]2

1 I1 1 dO + I1 on 1 f"'n on+2 dO - 2t(Yn) dO. "'n"'n

1

(1'+ 1

(26)

Equation (26) is a quadratic equation in t(); this quadratic equation assumes its minimum for

/III
We note that the Bayes estimators derived in Examples 43 to 45 are functions of sufficient statistics. It can be shown that this is generally true; that is, a Bayes estimator is always a function of minimal sufficient statistics. In fact, under quite general conditions it can be shown that the Bayes estimator corresponding to an arbitrary prior probability density function, which is positive for all 0 belonging to e, is consistent and BAN. So, even if you do not know the correct prior distribution, a Bayes estimator has some desirable optimum properties. And if you do know the correct prior distribution and accept the criterion that a best estimator is one that minimizes average loss, then the Bayes estimator corresponding to the known prior distribution is optimum. Even in those problems when the prior distribution is unknown, the concept of Bayes estimation can benefit us. It provides us with a technique of determining many estimators that we might not have otherwise considered. Each possible prior distribution has a corresponding estimator, whose merits can be judged by using our standard methods of comparison. Thus, we have yet another method of finding estimators to append to the methods given in Sec. 2. Bayes estimation can also sometimes be useful as a tool in obtaining an estimator possessing some desirable property that does not depend on prior distribution information. The property of minimax is such a property, and in the next subsection we will see how Bayes estimation can sometimes be used to find a minimax estimator. Another such property is given below. Our objective has been to minimize risk, but since risk depended on the parameter, we were unable to find one estimator that had smaller risk than all others for all parameter values. Minimax circumvented such difficulty by replacing the risk function by its maximum value and then seeking that estimator which minimized such maximum value. Another way of getting around the difficulty arising from attempting to uniformly minimize risk is to replace the risk function

VII

by the area under the risk function and to seek that estimator which has the least area under its risk function. We note that if the parameter space is an interval, the estimator having the least area under its risk function is the Bayes estimator corresponding to a uniform prior distribution over the interval e. This is true because for a uniform prior distribution the Bayes risk is proportional to the area under the risk function, and hence minimizing the Bayes risk is equivalent to minimizing area.

7.3

Minimax Estimator

We defined a minimax estimator at the end of Subsec. 3.4 as an estimator whose maximum risk is less than or equal to the maximum risk of any other estimator. Such an estimator might be considered "conservative" since it protects against the worst that can happen; it seeks to minimize the maximum risk. The following theorem is sometimes useful in finding a minimax estimator.

Theorem 14 If T*
PROOF

= 1*( Xl' ... , Xn) is a Bayes estimator having con-

stant risk, that is, Btt-(O)

= constant, then T* is a minimax estimator.

Let g*( . ) be the prior density corresponding to the Bayes estimator 1*( . , ... , .). sup Btt*(O) = constant = Bt,*(O)
6e~

f Btr<.O)g*(O) dO < J~ r BttCO)g*(O) dO ~ sup BtiO) e~

fi
6

for any other estimator I( . , ... , ').

IIII

EXAMPLE 46 Find the minimax estimator of 0 in sampling from the Bernoulli distribution using a squared-error loss function. We seek a Bayes estimator with constant risk. The family of beta distributions is a family of possible prior distributions. We hope that for one of the beta prior distributions the corresponding Bayes estimator will have constant risk. A Bayes estimator is given by
J~OOE X'(l
J~OEX'(l

- Ot- EX'[l IB(a, b)]oa-l(l - O)b-l dO - o)n-Ex'[l/B(a,b)]O a-l(l - O)b-l dO JAO Ex,+Q(1 - O)"-EXI+b-l dO = JAOExl+a-l(l - o)n-Ex,+b-l dO

_ BeL Xi + a + 1, n - 2: Xi + b) _ 2: Xi + a - B(2: + a, n - 2: + b) - n + a + b .
Xi Xi

VECTOR OF PARAMETERS

351

So -the- Bayes estimator with respect to a beta prior distribution having parameters -a and b is given by

We now ev8Juate the risk of (2: Xl + a)/(n + a + b) with the hope that we will be able to select a and b so that the risk will be constant. Write t': B(Xl, .. , x.) = A 2: Xi + B = (2: Xl + a)/(n + a + b); then t!it*A,B(O) = '4 [(A 2: Xi B - O?] = 8[[A(2: Xi - nO) + B - 0 + nAO]2] = A 24[(2: Xi - nO)2] + (B - 0 + nAO)2 = nA 20(l - 0) + (B - 0 + nA0)2 = 02[(nA - 1)2 _ nA2] + O[nA2 + 2(nA - I)B] + B2, which is constant if (nA - 1)2 - nA2 = 0 and nA2 + 2(nA - I)B = O. Now (nA - 1)2 - nA2 =0 if A=1/jn(jn+1), and nA2+2(nA-l)B=0 if B= -nA 2/2(nA - 1), which is 1/2(Jn + 1) for A = l/j;t(J~ + 1). On solving for a and b, we obtain a = b = j~/2; so (2: Xl + ~/2)/( n + is a Bayes estimator with constant risk and, hence, minimax. // //

In)

8 VECTOR OF PARAMETERS
In this section we present a brief introduction to the problem of simultaneous point estimation of several functions of a vector parameter. We will assume that a random sample Xl' ... , Xn of size n from the density I(x; 01, ... , Ok) is available, where the parameter 0 = (01, ... , Ok) and parameter space e are k-dimensional. We want to simultaneously estimate 't'l (0), ... , 't'r(O) , where 't'iO ) , j = I, ... , r, is some function of 0 = (01, ... , Ok)' Often k = r, but this need not be the case. An important special case is the estimation of 0 = (81, ... , e k) itself; then r = k, and 't'l (0) = 01, ... , 't' k(O) = Ok' Another important special case is the estimation of 't'(0); then r = 1. A point estimator of ('t'l(O), ... , 't',.(O is a vector of statistics , say(Tl" .. , T,.), where Tj = tiXl" .. , Xn) and Tj is an estimator of 't'j(0). Our presentation of the method of moments and maximum-likelihood method as techniques for finding estimators included the possibility that the parameter be vector-valued. So we already have methods of determining estimators. What we need are some criteria for assessing the goodness of an estimator, say (Th ... , T,.), and for comparing two estimators, say (Tl' ... , T,.) and (T{, ... , T;). As was the case in estimating a real-valued function 't'(e) , where we wanted the values of our estimator to be close to 't'(0), we now want the values of the estimator (1;., ... , Tr) to be close to ('t'l (0), ... , 't'r(O. We want the dis~ribution of (Tit ... , T,.) to be concentrated around ('t'1(0), ... , 't'r(O.

352

pAR.AMTlUC POINT ESTIMATION

VII

There are a number of ways of measuring the closeness of an estimator. For instance, in comparing two estimators the definitions of " more concentrated" and" closer," given in Subsec. 3.1, can be generalized to r dimensions. We will, however, restrict ourselves to consideration of unbiased estimators and define several ways of measuring the closeness of an unbiased estimator. No attempt will be made in this book to generalize to r dimensions the notions of loss and/or risk, invariance, Bayes estimation, and minimax. As far as optimum estimation is concerned, we will be content to consider only unbiased estimators and look for a best estimator within the restricted class of unbiased estimators.

Definition 33 Unbiased An estimator (T1 ) , T,.), where T j = t lX1 , , X,.}, j = 1, ... , r, is defined to be an unbiased estimator of (r1 (0), ... , 't,.(0) if and only if 4 e[1jl = 't/8) for j = 1, ... , r and for all Oee. /1//
In Sec. 5, where we considered unbiased estimation of a real-valued function 'teO), we employed the variance of an estimator as a measure of its closeness to't(O). Here we seek a generalization of the notion of variance to r dimensions. Several such generalizations have been proposed; we will consider four of them, called (i) vector of variances, (ii) linear combination (with nonnegative coefficients) of variances, (iii) ellipsoid of concentration, and (iv) Wilks' generalized variance. The last two require some knowledge of matrices. Possibly the simplest way of generalizing the concept of variance to r dimensions is to use the vector of variances of the unbiased estimators 11, ... , T,.. That is, let the vector (vare [Td, .,., varo [T,,]) be a measure of the closeness of the estimator (T1' ... , 1;.) to (-r1 (0), ... , t,,(O. The disadvantage of such a definition is that our measure is vector-valued and consequently sometimes difficult to work with. One way of circumventing this disadvantage is to use a linear combination of variances, that is, measure the closeness of the estimator (T1 , , T,.) to ('t1(0), ... , 't,.(0 with IJ=1 a j varo [1)] for suitably chosen aj > O. Both of these generalizations of variance embody only the variances of the Tj , j = I, ... , r. The Tj are likely to be correlated; so one might justifiably think that our measure of closeness of (T1' ... , 1;.) to ('t1 (8), ... , 't,.(0) should incorporate the covariances of the T/s.

Notation If (T1' .,., 1;.) is an unbiased estimator of ('t1(0), ... , 't,,(0, let uiiO) = cove [Tf, 1)]. The matrix whose ijth element is Ujj(O) is called the covariance matrix of the estimator (T1' ... , 1;.). Let uU(O) denote the ijth element of the inverse of the covariance matrix. / // /

VECTOR OF pARAMETERS

353

FIGURE 8

~----------------------~tl

Definition 34 Ellipsoid of concentration Let (T1' ... , T,.) be an un biased estimator of (r1 (0), ... , 'r,.(0)). Let uii(O) be the ijth element of the inverse of the covariance matrix of (T1' ... , T,.), where the ijth element of the covariance matrix is Ui/(/) = cOVo [T., Tj ]. The ellipsoid of concentration of (T1' . , T,.) is defined as the interior and boundary of the ellipsoid

i= 1

L J= L uii(O)[ti 1

,.

1:iCO )][tj - 'riO)] =

r+ 2.

(27)

See Fig. 8 for r

= 2.

1111

Loosely speaking, the ellipsoid of concentration measures how concentrated the distribution of (T1' ... , T,.) is about (1:1(0), ... , 1:,(0. [In fact, if one considers the vector random variable, say (U1 , . , U,), unifofI!1ly distributed over the ellipsoid of concentration, it can be proved that (U1 , , U,) and ('11, ... , T,.) have the same first- and second-order moments.] The distribution of an estimator (T1' ... , T,.) whose ellipsoid of concentration is contained within the ellipsoid of concentration of another estimator (T{, ... , T:) is more highly concentrated about ('r1 (0), ... , 1:,(0) than is the distribution of (T{, ... , T;). It is known that the determinant of the covariance matrix of an estimator is proportional to the square of the volume of the corresponding ellipsoid of concentration; hence another generalization of variance is as in Definition 35. Definition 35 Wilks' generalized variance Let ('11, .,., T,.) be an unbiased estimator of ('l'1 (0), ... , 1:,.(0). Wilks' generalized variance of (T1' ... , T,.) is defined to be the determinant of the covariance matrix of (T1""" T,.). 1111

354

pARAMETRIC POINT ESTIMA110N

VII

Theorem 8, which showed how sufficiency could be used to improve on an arbitrary unbiased estimator, generalizes to r dimensions. The generalization is stated without proof.

Theorem 15 Let Xl' ... , Xn be a random sample from the density

I(x; Ob ... , Ok), and let Sl = d l (Xl , ... , Xn), ... , Sm = d m (Xl , .,., X,J be a set of jointly sufficient statistics. Let (1";., . , T,.) be an unbiased estimator of ('rl (0), ... , 'rlO. Define Tj = 8[Tj ISl' ... , Sm], j = 1, ... , r. Then,
(i) (T{, ... , T;) is a statistic and an unbiased estimator of ('r l (O), ... , 'tr(O)), and T; = tj(Sb ... , S,J; that is, Tj is a function of the

sufficient statistics Sl' ... , Sm,j = 1, ... , r. (ii) varo [Ti] < varo [Tj ] for every 0 E i:J, j = 1, ... , r. (iii) The ellipsoid of concentration of (T{, ... , T;) is contained in the ellipsoid of concentration of (1";., .. , T,.), for every (J E e. //// We might note that (ii) implies that
aj j= 1

varo [Tn <

)= 1

L aj varo [Tj ]

and (iii) implies that Wilks' generalized variance of (T{, ... , T;) is smaller than Wilks' generalized variance of (Th ... , T,.). Theorem 10 of Sec. 5 can also be generalized to r dimensions, but first the concept of completeness has to be generalized.

Definition 36 Joint completeness For Xl' ... , X n, a random sample from the density I(x; 01 , , Ok), let (1";., , Tm) be a set of statistics. Tl , . .. , Tm are defined to be jointly complete if and only if 8 0 [,-,;::(1";., , Tm)] = 0 for all 0 E i:J implies that Po[.-.;::(1";., . , Tm) = 0] == 1 for all 0 E i:J, where .-.;::(1";., .. , TJ is a statistic. // / /

EXAMPLE 47

Let Xl' ... , Xn be a random sample from

Write 0 = (81 , 82 ), Let Yl = min [Xh ... , X,.] and where 81 < O 2 , Y,. = max [Xl' ... , Xn]. We want to show that Yi and Yn are jointly

VECTOR OF PARAMETERS

355

complete. (We know that they are jointly sufficient.) Let ~(Yl' Y,,) be an unbiased estimator of 0, that is,
<9' 8[~(Yl' Y,,)]

== 0

for all 0 E ~.

Now

which is identically 0 If and only if

f81

8 fYn~(yl' y")(y,, 2

Yl)

,,-2_

dYl dy,. = 0

81

Differentiate both sides with respect to O 2 , and obtain

now differentiate both sides of the resulting identity with respect to Oh and obtain - ~(01' ( 2 )(0 2 - 01ya-2 == 0 for all 01 < O 2 , and hence ~(01' ( 2 ) = 0 for 01 < O 2 ; that is, ~(Yl' YII) = 0 for Yl < Y,., where Yl and y,. are the possible values of Y We have shown that Y 1 and Y,.. 1 and Y,. are jointly complete. / // / If the density /(x; Oh ... , Ok) is a member of the k-parameter exponential family, a'set of jointly complete and sufficient statistics can be found using the following theorem. It is a k-dimensional analog of Theorem 9 and is stated without proof. The following theorem is not precisely stated; certain regularity conditions are omitted [16].

Theorem 16 Let Xl' ... , X,. be a random sample from/(x; 01 ,

If/(x; 01 , . is,/(x; 01 ,
... ,
"

Ok)'

Ok) =a(OI, ... , 0k)b(x)exp

[I c/0 1 , j=1

.,"

Ok) d.(x)j, that

J

Ct.

Ok) is a member of the k-parameter exponential family, then

d. (X,) ...

.t.

d.( Xi) ) is a minimal set of jointly complete and sufficient

statistics.

// //

3S6

pARAMETRIC POINT ES'I1MATION

VII

EXAMPLE 48

Let Xl' ... , Xn be a random sample from

Now

so

L Xi
i=l

and

L X;
i=l

are jointly complete and sufficient statistics by

Theorem 16.

IIII

We will state without proof the vector analog of Theorem 10. In the same sense that an UMVUE was optimum, this following theorem gives an optimum estimator for a vector of functions of the parameter.

Theorem 17 Let Xl' ... , Xn be a random sample from/ex; 01, ... , Ok)' Write 0 = (0 1, ... , Ok)' If Sl = 01(X1, ... , X n), ... , Sm = 0m(X1, ... , Xn) is a set of jointly complete sufficient statistics and if there exists an unbiased estimator of (r 1 (0), ... , rr(O, then there exists a unique unbiased estimator of (r1(0), ... , rr(O) , say Tt = tt(Sl' ... , Sm), ... , T: = t:(Sl' ... , Sm), where each tj is a function of Sl' ... , Sm, which satisfies:
var9 [Tt] ~ var 9 [1)] for every () E E:j, j = 1, ... , r, for any unbiased estimator (11, ... , T,) of (r 1 ((}), .. , r,((}. (ii) The ellipsoid of concentration of (Tt, ... , T:) is contained in the ellipsoid of concentration of (11, ... , 7;.), where (T1' ... , T,) is any II/I unbiased estimator of (r1 ((}), , rr(}'
(i)

There are four different maximal subscripts, all of which are intended. n denotes the sample size, k denotes the dimension of the parameter 0, m is the number of real-valued statistics in our jointly complete and sufficient set, and r is the dimension of the vector of functions of the parameter that we are trying to estimate. In practice, it will turn out that usually' k = m. The estimator (Tt, ... , T:) is optimal in the sense that among unbiased estimators it is the best estimator using any of the four generalizations of variance that have been proposed. Just as was the case in using Theorem 10, we have two ways of finding (Tt, ... , r:). The first is to guess the correct form of the functions tt, ... , which are functions of Sl' ... , Sm' that will make them unbiased estimators of

t:,

VECTOR OF PARAMETERS

357

't1 (0), ... , 'f,(l1). The second is to find any set of unbiased estimators of 'fl(O), , 'f,i.l1) and then calculate the conditional expectation of these unbiased estimators given the set of jointly complete and sufficient statistics. We employ only the first method in the following examples.

EXAMPLE 49 Let Xi) ... , Xn be a random sample from the density f(x; 01 , ( 2 ) = [1/(0 2 - ( 1 )]/(81. (2)(X)' Suppose we want to jointly estimate the range and midrange, that is, 't1 (8) = 82 - 01 and 't2(0) = (0 1 + ( 2 )/2. We know that Yi = min [Xl' . " Xn] and Y n = max [Xi) , .. , Xn] are jointly sufficient (see Example 23); also, they are jointly complete (see Example 47). Hence, to find the unbiased estimator (T!, Tn which has uniformly smallest variance for each component among all unbiased estimators, it suffices to find the unbiased estimator that is a function of the jointly complete sufficient statistics. Since 4[Y1]= 01+ (0 2- (1)/(n+ 1) and 4[Yn] = O 2 - (82 - (1)/(n + 1), ([(n + 1}/(n - 1}](Yn- Y1 ), (Yl + Yn)/2) is the unbiased estimator of (0 2 - 01 , (01 + ( 2 )/2) that we are seeking. 111I

EXAMPLE 50 Let Xi) ... , Xn be a random sample from the normal density f(x; 01, ( 2 ) = tP/l.fl2(X). By Examples 22 and 48, L Xi and L Xl are jointly complete and sufficient statistics. Hence, by Theorem 17, (L X,ln, L (X, - X)2/(n - 1)) is an unbiased estimator of (Il, 0'2) whose corresponding ellipsoid of concentration is contained in the ellipsoid of concentration of any other unbiased estimator. [NoTE: L (X, - X)2 = L Xf - nX2; so the estimator L (Xi - X)2/(n - 1) is a function of the jointly complete and sufficient statistics Xi and X;,] For this same example, suppose we want to estimate that function of 8 = (p., 0'2) satisfying the following integral equation:

for (X fixed and known. 'teO) is that point which satisfies P[Xi > 'teO)] = (X; that is, it is that point which has 100(X percent of the mass of the population density to its right, or r(O) is the (1 - (X)th quantile point. We have 1 - (X = <b(['t(0) - 1l]10'); so 'teO) = Jl + ZI-O', where Zl- is given by )(Zl -J = 1 - (x, Since (X is known, Zl _ can be obtained from a table of the standard normal distribution. To find the LTMVUE of reO), it suffices to ~nd the unbiased estimator of Jl + Zl _ 0' which is a function of

3S8

PARAMETRIC POINT ESTIMATION

vu

L Xi

and L verified that

xl

We know that X is the UMVUE of /1, and it can be

r[(n - 1)/2] r(nI2)j2

J'L (Xi -

X)2

= T*

say, is the UMVUE of (1; hence X + Zl-a T* is the UMVUE of T(O). We have employed Theorem 17 for r = 1; our vector of functions of the parameter that we wanted to estimate was unidimensional. fill

OPTIMUM PROPER TIES OF MAXIMUM-LIKELIHOOD ESTIMATION

Several methods of finding point estimators were presented in Sec. 2 of this chapter. There, and in succeeding sections, we have particularly emphasized the method of maximum likelihood. In this section we will partially justify such emphasis by considering some optimum properties of maximum-likelihood estimators. For simplicity of presentation, let us consider the maximum-likelihood estimation of the parameter 0, which is to be estimated on the basis of a random sample from a density f( ; 0), where 0 is assumed to be a real number. That is, let us consider the unidimensional-parameter case and estimate 0 itself. Recall that for the observed sample Xl' ... , Xn the maximum-likelihood estimate of 0 is that value, say 0, of 0 which maximizes the likelihood function L(O; Xl' ... , Xn)

i= 1

nf(Xi;
n

0).

Let an

= 8iXl ,

... , Xn) denote the maximum-likelihood

estimator of 0 based on a sample of size n. We defined and discussed in Sec. 3 of this chapter a number of properties that an estimator mayor may not possess. Recall that some of these properties, such as unbiasedness and uniformly minimum variance, are referred to as small-sample properties, and others of these properties, such as consistency and best asymptotically normal, are referred to as large-sample properties. The use of the word" small" in "small-sample" is somewhat misleading since a small-sample property is really a property that is defined for a fixed sample size, which may be fixed to be either small or large. By a large-sample property, we mean a property that is defined in terms of the sample size increasing to infinity. Our main result of this section will be contained in Theorem 18 below and will concern optimum large-sample properties of maximum-likelihood estimation.

OPTIMUM PROPERTIES OF MAXIMUM-LIKELIHOOD ESTIMATION

359

We have already observed some small-sample properties of maximumlikelihood estimation. For instance, we have noted two things: first, that some maximum-likelihood estimators are unbiased and others are not and, second, that some maximum-likelihood estimators are uniformly minimum-variance unbiased and others are not. For example, in the density f(x; 8) = 4>6, I (x) the maximum-likelihood estimator of 8 is X, which is the uniformly minimumvariance unbiased estimator of 8, whereas in the density f(x; 8) = (1/8)/[0. 8](X) the maximum-likelihood estimator of 8 is Y,. = max [Xl' ... , X,.], which is biased. [We might note here that the Yn in this last example can be corrected for bias by multiplying Yn by (n + l)/n and that the estimator that is thus obtained is uniformly minimum variance unbiased.] One property that it seems reasonable to expect of a sequence of estimators is that of consistency. Theorem 18 will show, in particular, that generally a sequence of maximum-likelihood estimators is consistent. Theorem 18 If the density f(x; 8) satisfies certain regularity conditions and if An = 8n (Xh . , Xn) is the maximum-likelihood estimator of 8 for a random sample of size n from f(x; 8), then:
(i)

An is asymptotically normally distributed with mean 8 and

variance 1/114

9[[:0 logf(X; 6)]1

AI' ... , An'
/ ///

(ii) The sequence of maximum-likelihood estimators

... is best asymptotically normal (BAN).

We will not be able to prove Theorem 18. In fact, we have not precisely stated it, inasmuch as we have not delineated the regularity conditions. We do, however, want to emphasize what the theorem says. Loosely speaking, it says that for large sample size the maximum-likelihood estimator of 8 is as good an estimator as there is. (Other estimators might be just as good but not better.) We might point out one feature of the theorem, namely, that the asymptotic normal distribution of the maximum-likelihood estimator is not given in terms of the distribution of the maximum-likelihood estimator. It is given in terms of f( ; B), the density sampled. Also, the variance of the asymptotic normal distribution given in the theorem is the Cramer-Rao lower bound. EXAMPLE 51 Let Xl"'" Xn be a random sample from the negative exponential distribution f(x; 8) = Be-Ox/[o. a:(x). It can be routinely demonstrated that the maximum-likelihood estimator of B is given by

360

PARAMETRIC POINT ESTIMA nON

VB

nIL Xi = II X
1

Fa

Fa.

According to Theorem 18 above, the maximum-likeli-

hood estimator has an asymptotic normal distribution with mean 0 and variance equal to
1
-----------:~

n4.[[! 10gf(X; 0)] 2]

= -.

(See Example 29.)

IIII

We have ordinarily considered estimation of T(O) some function of 8, rather than estimation of 8 itself. For maximum-likelihood estimation, we noted (see Theorem 2) that the maximum-likelihood estimator of -r(8) was given by T(a), where a was the maximum-likelihood estimator of O. If we assume that T( . ) is differentiable, then it can be shown that T(a) has an asymptotic normal distribution with mean T(O) and variance

"88 [[oologf(X;

[T'(8)]2

Ii)] ]
2

'

which is the Cramer-Rao lower bound. (See Theorem 7.) Maximum-likelihood estimators possess similar optimum large-sample properties in the case of a k-dimensional parameter. For instance, it can be proved (again under regularity conditions) that the joint distribution of the maximum-likelihood estimators is asymptotically distributed as a multivariate normal distribution. Let us illustrate for the case when k = 2; that is, 8 = (0 1 , 82 ). Recall that the bivariate normal distribution is specified by the five parameters /1h /12, ui, ui, and p. (See Sec. 5 of Chap. IV.) It tUrns out that under certain regularity conditions the joint distribution of the maximumlikelihood estimators a1 and z is asymptotically distributed as a bivariate normal distribution with parameters /11 = 81, /12 = 82 ,

-.t.[::~IOgf(X; 0)]
(J2 -1

n~

2 u2 -

-.t'[::i logf (X; 0)]

nA

oPTIMUM PROPERTIES OF MAXIMUM-LIKELlHOOD ESTIMATION

361

and

PU 1U 2

nA

where

A = .,.[::~ log/(X; 9)]".[::~ log/(X; 9)] -

(8.[00::9, log/(X; 9)])'.

e-(1/ 20 z)(x-0 1 )Z
2

EXAMPLE 52 Let Xl' .. , X,. be a random sample from the density

f( x', ll) v

= f(x', 0H

8)
2

= '+'01.Oz (x) = J2n8 1

,I,.

We have already derived, in Example 6, the maximum-likelihood estimators of 01 and 82 ; they are, respectively,

G1 = -

nf '

~ X

II

and

According to the above, the asymptotic large-sample joint distribution of G1 and G2 is a bivariate normal distribution with means 01 and O 2 Since 10gf(X; 0) = -t log 2n - t log 8 2 - (l/20 2 )(X - ( 1)2, the required deri vatives are

82 I 80 2 logf( X; 0) = - ll'
1

u2

80 80 10gf(X; 0) = 2 1

82

X - 01

02
2

'

and

and because

362

pARAMETRIC POINT ESTIMATION

vn

and

-&.[::~ logf(X; II)] = ~r

which gives A

= 1/20~.

Finally, then,

ui = 021n, u~ = 20iln, and p = o.

IIII

PROBLEMS
1

An urn contains black and white balls. A sample of size n is drawn with replacement. What is the maximum-likelihood estimator of the ratio R of black to white balls in the urn? Suppose that one draws balls one by one with replacement until a black ball appears. Let X be the number of draws required (not counting the last draw). This operation is repeated n times to obtain a sample X., X 2 , , X n What is the maximum-likelihood estimator of R on the basis of this sample '/ Suppose that n cylindrical shafts made by a machine are selected at random from the production of the machine and their diameters and lengths measured. It is found that Nil have both measurements within the tolerance limits, Nl2 have satisfactory lengths but unsatisfactory diameters, N21 have satisfactory diameters but unsatisfactory lengths, and N22 are unsatisfactory as to both measurements. N'J = n. Each shaft may be regarded as a drawing from a multinomial popUlation with density

2:

PH P12 P21

~ll

~12

~21(l

-PH - Pl2 - P21

)~22

for

X'J

= 0, l; 2: Xu = 1

having three parameters. What are the maximum-likelihood estimates of the parameters if Nu = 90, N12 = 6, N21 = 3, and N22 = 1 '/ 3 Referring to Prob. 2. suppose that there is no reason to believe that defective diameters can in any way be related to defective lengths. Then the distribution of the X,Jcan be set up in terms of two parameters: PI, the probability of a satisfactory length, and q., the probability of a satisfactory diameter. The density of the Xu is then
(Plqlt ll [Pl(1 - ql)t 12 [(1 - PI)qlf 2I [(l - Pl)(l - ql)t

22

for

XIJ

=0, l;

2: X'J = 1.

What are the maximum-likelihood estimates for these parameters'/ Are the probabilities for the four classes different under this model from those obtained in the above problem '/ 4 A sample of size nl is to be drawn from a normal population with mean P,l and variance A second sample of size n2 is to be drawn from a normal population with mean P,2 and variance O'~. What is the maximum-likelihood es~timator of Jl~_ h () = P,l - P,2 '/ If we assume that the total sample SIze n = nl + n2 IS xed, Ow should the n observations be divided between the two populations in order to minimize the variance of the maximum-likelihood estimator of () '/

0':.

PROBLEMS

363

5 A sample of size n is drawn from each of four normal populations. all of which have the same variance a 2 The means of the four populations are a + b + c, a + b - c, a - b + c, and a - b - c. What are the maximum-likelihood estimators of a, b, c, and a 2 ? (The sample observations may be denoted by X'J, i = 1, 2, 3,4 andj = 1,2, ... , n.) 6 Observations X., X 2 , , Xn are drawn from normal populations with the same mean ft but with different variances a~, a~, ... ,a;. Is it possible to estimate all the parameters? If we assume that the a~ are known, what is the maximumlikelihood estimator of ft ? 7 The radius of a circle is measured with an error of measurement which is distributed N(O, ( 2 ), a 2 unknown. Given n independent measurements of the radius, find an unbiased estimator of the area of the circle. 8 Let X be a single observation from the Bernoulli density f(x; (J) = (Jx(1- (J)l-xI{o.l)(X), where 0 < (J < 1. Let tl(X) = X and t 2(X) = l. (a) Are both tl(X) and t 2(X) unbiased? Is either? (b) Compare the mean-squared error of tl(X) with that of t 2(X). 9 Let Xl, X 2 be a random sample of size 2 from the Cauchy density
f(x, ) - 17[1
.(J _

+ (x _

8)2]'

- 00 <

(J

< 00.

Argue that (Xl + X 2 )/2 is a Pitman closer estimator of (J than Xl is. [Note that (Xl + X 2 )/2 is not more concentrated than Xl since they have identical distributions.] 10 Let (J denote some physical quantity, and let Xl, , Xn denote n measurements of the physical quantity. If (J is estimated by 0, then the residual of the ith measurement is defined by X, - 0, i = 1, "', n. Show that there is only one estimator with the property that the residuals sum is 0, and find that estimator. Also, find that estimator which minimizes the sum of squared residuals. II Let Xl, , Xn be a random sample from some density which has mean ft and variance a 2
(a)

Show that .2 a, X, is an unbiased estimator of ft for any set of known constants

1

al, .. , an satisfying

.2 a, =
1

1.

(b)

If

L a, =
1 -

1, show that var

n

[L a, X,] is minimized for a, =

1
n

lIn, ; = 1, ... , n.

[HINT: Prove that

L a: = L (a, 1 1

I/n)2 + lIn when

~ a, = f

1.]

12 Let X., , Xn be a random sample from the discrete density function f(x; 8) =
(Jx(1 - (J)l-xI{o. l)(X)~ where 0 :::;;: (J
(a)

< l. Note that e =

{(J: 0 < (J

< n.

Find a method-of-moments estimator (J, and then find the mean and meansquared error of your estimator. (b) Find a maximum-likelihood estimator of (), and then find the mean and mean-squared error of your estimator.

364

pAR.AMJ!TR.IC poINT ESTIMATIoN

VB

13 Let Xl, X 2 be a random sample of size 2 from a normal distribution with mean 8

and variance 1. Consider the following three estimators of 8:

T, = t,(Xl, X 2 ) = IX,
T2

+ tX2

= t 2 (X" X 2 ) = IX, + iX2 T3 = t 3 (X" X 2 ) = IX, + IX2

For the loss function t(t; 8):::;;: 38 2 (t - 8)2, find 1t,,(8) for i = 1, 2, 3, and sketch it. (b) Show that T, is unbiased for i = 1, 2, 3. 14 Let X" ... , XII: .. be independent and identically distributed random variables from some distribution for which the first four central moments exist. We know that 8[8 2 ] 0'2 and
(a)

1 ( P,4 - n3 ) var [8 2 ] = - 0'4 , n n-l where

Is 8 2 a mean-squared-error consistent estimator of 0'2 ? 15 In genetic investigations one frequently samples from a binomial distribution
f(x) =

(7)

p'q'" -. except that observations of x = 0 are impossible; so, in fact, the

sampling is from the conditional (truncated) distribution

m) yqm-x ( qm I{l.
X _

m}(x).

Find the maximum-likelihood estimator of p in the case m = 2 for samples of size n. Is the estimator unbiased ? 16 Let X be a single observation from N(O, 8). (8 = 0'2.) (a) Is X a sufficient statistic? (b) Is I XI a sufficient statistic? (c) Is X2 an unbiased estimator of 8? (d) What is a maximum-likelihood estimator of V 8? (e) What is a method-of-moments estimator of V 8? 17 Let X have the density f(x;8) = (8/2)lxl(1- 8)'-l x II{_,. o.,}(x), o:s;: 8 <1. Define t(x) = 2/{l)(x). (a) Is X a sufficient statistic? A complete statistic? (b) Is IXI a sufficient statistic? A comp~e statistic? (c) What is a maximum-likelihood estimator of 8? (d) Is T= t(X) an unbiased estimator of 8? (e) Does f(x; 8) belong to an exponential class ? (f) Find an estimator with uniformly smaller mean-squared error than that of t(X), if such exists.

PROBLEMS

365

18 Let Xh X 2 ,

Xn be a random sample from the density

I(x; 8) = ()x- 2 /(6. CIJ)(x)

where 8>0. (a) Find a maximum-likelihood estimator of (). (b) Is YI = min [X., . , Xn] a sufficient statistic? 19 Let Xl, , XII be a random sample from I(x; 8) = Ie -Ix - 61, - 00 < () < 00. (a) Discuss sufficiency for this density. (b) Obtain a method-of-moments estimator of (). (e) Find a maximum-likelihood estimator of (). (d) Does I(x; 8) belong to an exponential class? 20 Find a maximum-likelihood estimator for oc in the density I(x; oc) = (2/oc 2 )(oc-- x)/eo )(x) for samples of size 2 Is it a sufficient statistic? Estimate oc by the method of moments. What is the maximum-likelihood estimator of the population mean? 21 Let Xl, . , Xn be a random sample from I(x; 8) = (1/8)/(0. 61(X), where () > O. Define Yn = max [X., .. , Xn] and Yl = min [X., ... , X n] (a) Estimate () by the method of moments. Call the estimator T l Find its mean and mean-squared error. (b) Find the maximum-likelihood estimator of (). Call the estimator T 2 Find its mean and mean-squared error. (e) Among all estimators of the form a Y n , where a is a constant which may depend on n, find that estimator which has uniformly smallest mean-squared error. Call it Ts. Find its mean and mean-squared error. (d) Find the UMVUE of (). Call it T 4 Obtain its mean and mean-squared
II

Let Ts = Yl + Yn Find the mean and mean-squared error of Ts. (I) What estimator of () would you use and why? (g) Find the maximum-likelihood estimator of the variance of the population. 22 Let Xl, . .. , Xn be a random sample from the Bernoulli distribution, say P[X = 1] =
(e)

error.

() = 1 - P[X = 0].
(a)

Find the Cramer-Rao lower bound for the variance of unbiased estimators of
()(1 - 8).

Find the UMVUE of ()(1 - 0) if such exists. 23 Assuming r known, find the maximum-likelihood estimator for" for a random sample of size n from a gamma distribution. Find a sufficient statistic if one exists,,\ Is your maximum-likelihood estimator unbiased? Is there an UMVUE of 24 Let X., ... , Xn be a random sample from ()x 6 -1leo. 1)(x), where () > O. (a) Find the maximum-likelihood estimator of t-t = ()/(1 + ()). (b) Find a sufficient statistic, and check completeness. Is L: X, a sufficient statistic? (e) Is there a function of () for which there exists an unbiased estimator whose variance coincides with the Cramer-Rao lower bound? *(d) Find the UMVUE of each of the following: (), 1/(), t-t = ()/(l + 8).
(b)

,,?

366

pARAMETRIC POINT ESTIMATION

VII

25

Let X., . , X,. be a random sample from the binomial distribution

(:)P"(I-

p)m-., x

~ 0,1, . , m, where m is known and 0 <p< 1.

Estimatep by the method of moments and the method of maximum likelihood. Is there an UMVUE of p? If so, find it. *26 Let Xl, ... , Xn be a random sample from the discrete density function
(a) (b) f(x; 8)

= (1/8)lu . 2 9) (X) ,

where () = 1, 2, .... That is, = {(): () = 1, 2, ... } = the set of positive integers. (a) Find a method-of-moments estimator of (). Find its mean and mean-squared error. (b) Find a maximum-likelihood estimator of (). Find its mean and mean-squared error. (c) Find a complete sufficient statistic. (d) Let T = Yn , the largest order statistic. Show that the UMVUE of () is
[Tn+1 - (T-1)n+1]/[T" - (T- 1)n].

27 Let X be a single observation from the density [l/B(), 8)]X 9- 1(1- x)9- 11(0. 1)(X). Is X a sufficient statistic? Is X complete? 28 An experimenter knows that the distribution of the lifetime of a certain component is negative exponentially distributed with mean 1/(). On the basis of a random sample of size n of lifetimes he wants to estimate the median lifetime. Find both the maximum-likelihood and uniformly minimum-variance unbiased estimator of the median. 29 Let Xl, . , Xn be a random sample from N(), 1). (a) Find the Cramer-Rao lower bound for the variance of unbiased estimators of (), ()2, and p[X > 0]. (b) Is there an unbiased estimator of ()2 for n = ]? If so, find it. (c) Is there an unbiased estimator of P[X > O]? If so, find it. (d) What is the maximum-likelihood estimator of P[X > O]? (e) Is there an UMVUE of ()2? If so, find it. (f) Is there an UMVUE of P[X > O]? If so, find it. 30 For a random sample from the Poisson distribution, find an unbiased estimator of 7(A) = (1 + A)e- A Find a maximum-likelihood estimator of T(A). Find the UMVUE of T(A). 31 Let X., .. , Xn be a random sample from the density
f(x; 8) = 2x

7f2 1(0.9)(x)

where () > O. (a) Find a maximum-likelihood estimator of (). v(b) Is Yn = max [Xl, , Xn] a sufficient statistic? Is Y n complete? (c) Is there an UMVUE of ()? If so, find it.

PROBLEMS

367

32 Let XI, , X" be a random sample from the density

f(x; U) = 0(1
(a)

+ x)-U +

9)/(0.

C()(x)

for 0 >0.

(b)
(c) (d) (e) (f)

33 Let

Estimate 0 by the method of moments assuming 0 > 1. Find the maximum-likelihood estimator of I/O. Find a complete and sufficient statistic if one exists. Find the Cramer-Rao lower bound for unbiased estimators of I/O. Find the UMVUE of I/O if such exists. Find the UMVUE of 0 if such exists. X h , X" be a random sample from
f(x; 0)

for 0

o.

Find a maximum-likelihood estimator of O. Suppose n 1, so that you have only one observation, say X = Xl' Clearly X is a sufficient statistic. Is X a minimal sufficient statistic? Is X complete? 34 Let Xl, . , X" be a random sample from the negative exponential density
(a) (b)

f(x; 0) = Oe-h/[o. C()(x).

Find the uniformly minimum-variance unbiased estimator of var [Xl] if such exists. (b) Find an unbiased estimator of I/O based omy on Y~") = min [Xl, ... , XJ. Is your sequence of estimators mean-SQuared-error consistent? 35 Let Xl, ~ . , X" be a random sample from the density
(a)

log 0 f(x; 0) = 0 _ 1 OX/(o.l)(x),

(a) (b)

0>1.

Find a complete sufficient statistic if there is one. Find a functiOn of 0 fQr which there exists an unbiased estimator whose variance coincides with the Cramer-Rao lower bound if such exists. 36 Show that

.g. [{

:0 log/(X;

6)

= - 8. [ :;. log/(X; 6)

37 Let Xl, "', X" be a random sample from the density

f(x; U) = e-(x-B) exp (_e-(X-B,

where - 00 < 0 < 00. (a) Find a method-of-moments estimator of O. (b) Find a maximum-likelihood estimator of O. (c) Find a complete sufficient statistic. (d) Find the Cramer-Rao lower bound for unbiased estimators of O. (e) Is there a function of 0 for which there exists an unbiased estimator, the variance of which coincides with theCramer-Rao lower bound? If so, find it. *(f) Show that r'(n)/r(n) - log (2: e- Xt ) is the UMVUE of O.

368

pARAMBTRIC POINT ESTIMATION

vn

38 Let X., , Xn denote a random sample from

I(x; fJ) = 19(x)

= fJ/l(x)

(1 - fJ)/o(X),

39

40

41

42

where 0 fJ < 1 andh(') and/o(') are known densities. (a) Estimate fJ by the method of moments. (b) For n = 2, find a maximum-likelihood estimator of fJ. (c) Find the Cramer-Rao lower bound for the variance of unbiased estimators of fJ. Suppose that a(') and b(') are two nonnegative functions such that I(x; fJ) = a(fJ)b(x)/(o.9)(X) is a probability density function for each fJ > O. (a) What is a maximum-likelihood estimator of fJ? (b) Is there a complete sufficient statistic? If so, find it. (e) Is there an UMVUE of fJ? If so, find it. Let Xl, , Xn be a random sample from N(fJ, fJ), fJ > O. (a) Find a complete sufficient statistic if such exists. (b) Argue that Xis not an UMVUE of fJ. (e) Is fJ either a location or scale parameter? Let Xl, .. , Xn be a random sample from N(fJ, fJ2), - 00 < fJ < 00. (a) Is there a unidimensional sufficient statistic? (b) Find a two-dimensional sufficient statistic. (e) Is Xan UMVUE of fJ? {HINT; Find an unbiased estimator of fJ based on 8 2 ; call it T*. Find a constant a to minimize var [aX + (1 - a)T*].} (d) Is fJ either a location Or scale parameter? Let Xl, ... , Xn be a random sample of size n from the density
I(x; fJ)
(a) (b) (e)

= -0 1[9. 29](X),

fJ

> O.

Find a maximum-likelihood estimator of fJ. We know that Y1 and Yn are jointly sufficient. Are they jointly complete? Find the Pitman estimator for the scale parameter fJ. (d) For a and b constant (they may depend on n), find an unbiased estimator of fJ of the form a YI b Y n satisfying P[ Y n/2 < a YI b Yn < Yd = 1 if such exists. Why is P[Yn/2 < aYI bY" YI ] = 1 desirable? 43 Let Z 1, , Z" be a random sample from N(O, fJ2), fJ > O. Define X, = IZ, I" and consider estimation of fJ and fJ2 on the basis of the random sample Xl, ... , X". (a) Find the UMVUE of fJ2 if such exists. (b) Find an estimator of fJ2 that has uniformly smaller mean-squared error than the estimator that you found in part (a). (c) Find the UMVUE of fJ if such exists. (d) Find the Pitman estimator for the scale parameter fJ. (e) Does the estimator that you found in part (d) have uniformly smaller meansquared error than the estimator that you found in part (e)?

PROBLEMS

369

44 Let XI, , XII be a random sample from

f(x; 0) =
e-(X-6)I[6.

o:(x)

for -

00

< () <

00.

(a) Find a sufficient statistic. (b) Find a maximum-likelihood estimator of 0. (c) Find a method-of-moments estimator of 0. (d) Is there a complete sufficient statistic? If so, find it. (e) Find the UMVUE of () if one exists.

Find the Pitman estimator for the location parameter (). Using the prior density g(O) = e- 6 I(o. 0:(0), find the posterior Bayes estimator of 0. 45 Let Xl, , X" be a random sample from f(xI8) 8x 6 - 1 I(o.1)(x), where e > o. Assume that the prior distribution of e is given by
(f) (g)

where r and A are known. (a) What is the posterior distribution of e? (b) Find the Bayes estimator of () with respect to the given gamma prior distribution using a squared-error loss function. 46 Let X be a single observation from the density f(xl 0) = (2x/()2)I(o. ,)(x), where () > O. Assume that e has a uniform prior distribution over the interval (0, 1). For the loss function (t; 0) ()2(t - 0)2, find the Bayes estimator of 0. 47 Let Xl, X 2 , , XII be a random sample of size n from the follOWing discrete density:

where () > O. (a) Is there a unidimensional sufficient statistic? If so, is it complete? (b) Find a maximum-likelihood estimator of 0 2 = P[XI = 2]. Is it unbiased? (c) Find an unbiased estimator of () whose variance coincides with the corresponding Cramer-Rao lower bOund if such exists. If such an estimate does not exist, prove that it does not. (d) Find a uniformly minimum-variance unbiased estimator of ()2 if such exists. (e) Using the squared-error loss function find a Bayes estimator of 0 with respect to the beta prior distribution

(f) (g)

Using the squared-error loss function, find a minimax estimator of (). Find a mean-squared error consistent estimator of 02

370

PARAMETRIC POINT ESTIMATION

VII

48

Let Xl, ... , Xn be a random sample from a Poisson density

e-fJ(Jx

- , - 1,0. 1
X.

...

}(X),

where () > O. For a squared-error loss function find the Bayes estimator of () for a gamma prior distribution. Find the posterior distribution of 0. Find the posterior Bayes estimator of T( () = P[Xf = 0]. 49 Let Xl, ... , Xn be a random sample from f(xl () = (1/()/(o.fJ)(x), where () > o. For the loss function (t - 8)2/()2 and a prior distribution proportional to ()-IZ/(1. CXI)() find the Bayes estimator of 8. 50 Let Xl, ... , Xn be a random sample from the Bernoulli distribution. Using the squared-error loss function, find that estimator of () which has minimum area under its risk function. 51 Let Xl, ... , Xn be a random sample from the geometric density
f(x; 8)

= ()(1 - ()"I,o.l . ... }(x)

where 0 < () < 1. (a) Find a method-of-moments estimator of (). (b) Find a maximum-likelihood estimator of (). (c) Find a maximum-likelihood estimator of the mean. (d) Find the Cramer-Rao lower bound for the variance of unbiased estimators of 1 - (). (e) Is there a function of () for which there exists an unbiased estimator the variance of which coincides with the Cramer-Rao lower bound? If so, find it. (f) Find the UMVUE of (1- ()/() if such exists. (g) Find the UMVUE of () if such exists. (h) Assume a uniform prior distribution and find the posterior distribution of 0. For a squared-error loss function, find the Bayes estimator of () with respect to a uniform prior distribution. 52 Let () be the true I.Q. of a certain student. To measure his I.Q., the student takes a test, and it is known that his test scores are normally distributed with mean fL and standard deviation 5. (a) The student takes the I.Q. test and gets a score of 130. What is the maximumlikelihood estimate of ()? (b) Suppose that it is known that I.Q.'s of students of a certain age are distributed normally with mean 100 and variance 225; that is, 0 ,.... N(100, 225). Let X denote a student's test score [X is distributed N(), 25)]. Find the posterior distribution of 0 given X = x. What is the posterior Bayes estimate of the student's I.Q. if X = 130. *53 Let Xl, ... , Xn be a random sample from the density

PROBLEMS

371

where - 00 < a < 00 and f3 o. Show that Yl and 2: Xi are jOintly sufficient. It can be shown that Yl and 2: (Xi - Yl ) are jOintly complete and independent of each other. Using such results, find the estimator of (a, f3) that has an ellipsoid of concentration that is contained in the ellipsoid of concentration of any other unbiased estimator of (a, (J). (Yl = min [Xl"'.' X n1.) 54 Let X., ... , X. be a random sample from the density
f(x; a, 8)

= (1- 8)(JX-I(<<, +1, ... )(x),

where - 00 < a < 00 and 0 < (J < 1. (a) Find a two-dimensional set of sufficient statistics. (b) Find the maximum-likelihood estimator of (J). 55 Let Xl, ... , Xn be a random sample from the density

<a,

f<x; 8) =

2x 2(1- x) e l(o.9}(x) + 1 _ 1(9.1](X),

(J

where 0 < (J < 1. (a) Estimate (J by the method of moments. (b) Find the maximum-likelihood estimator of (J for n = 1 and n = 2. (c) For n 1 find a complete sufficient statistic if such exists. Find a UMVUE of (J for n = 1 if such exists. *(d) Find the maximum-likelihood estimator of (J.

VIII
PARAMETRIC INTERVAL ESTIMATION

1 INTRODUCTION AND SUMMARY

Chapter VII dealt with the point estimation of a parameter, or more precisely, point estimation of a value of a function of a parameter. Such point estimates are quite useful, yet they leave something to be desired. In all those cases when the point estimator under consideration had a probability density function, the probability that the estimator actually equaled the value of the parameter being estimated was O. (The probability that a continuous random variable equals anyone value is 0.) Hence, it seems desirable that a point estimate should be accompanied by some measure of the possible error of the estimate. For instance, a point estimate might be accompanied by some interval about the point estimate together with some measure of assurance that the true value of the parameter lies within the interval. Instead of making the inference of estimating the true value of the parameter to be a point, we might make the inference of estimating that the true value of the parameter is contained in some interval. We then speak of interval estimation, which is to be the subject of this chapter. Like point estimation, the problem ofinterval estimation is twofold. First, there is the problem offinding interval estimators, and, second, there is the prob-

CONFIDENCE INTERVALS

373

lern of determining good, or optimum, interval estimators. The considerations of these two problems that will appear in this chapter will be incomplete. Further considerations will be presented at the end of the next chapter on testing hypotheses. The mathematics of interval estimation and hypotheses testing are closely related. Either concept could be used to introduce the other. In this book, we have decided to introduce interval estimation first, right after our presentation of point estimation, then introduce hypotheses testing, and finally point out the close mathematical relationship between the two. The introduction to interval estimation that appears in this chapter will not be as thorough as was our discussion of point estimation in the last chapter. One should not infer from this that interval estimation is less important since in practice the opposite is usually true. It is just easier to present the basic theory of point estimation. No concerted effort will be given to the problem of finding optimum interval estimators. The chapter will be divided into six main sections, the first being this introductory section. Section 2 will be devoted to confidence intervals, where the notion is introduced and defined. One method of finding confidence intervals will also be given as well as some idea as to what an optimum confidence interval might be. Section 3 will consider several examples of confidence intervals that are associated with sampling from the normal distribution. Such discussion will hinge on the results of Sec. 4 of Chap. VI. Several general methods of finding confidence intervals are given in Sec. 4; another method, which utilizes the theory of hypotheses testing, will be given at the end of Chap. IX. A brief discussion of large-sample confidence intervals appears in Sec. 5, and Sec. 6 presents another type of interval estimation, namely, Bayesian interval estimation.

CONFIDENCE INTERVALS

2.1

An Introduction to Confidence Intervals

In practice, estimates are often given in the form of the estimate plus or minus a certain amount. For instance, an electric charge may be estimated to be (4.770 + .005)10- 10 electrostatic unit with the idea that the first factor is very unlikely to be outside the range 4.765 to 4.775. A cost accountant for a publishing company in trying to allow for all factors which enter into the cost of producing a certain book (actual production costs, proportion of plant o,("erhead, proportion of executive salaries, etc.) may estimate the cost to be 83 + 4.5 cents per volume with the implication that the correct cost very probably lies between

374

PARAMETRIC INTERVAL ESTIMA nON

vm

78.5 and 87.5 cents per volume. The Bureau of Labor Statistics may estimate the number of unemployed in a certain area to be 2.4 + .3 million at a given time, feeling rather sure that the actual number is between 2.1 and 2.7 million. What we are saying is that in practice One is quite accustomed to seeing estimates in the form of intervals. In order to give precision to these ideas, we shall consider a particular example. Suppose that a random sample (1.2, 3.4, .6, 5.6) of four observations is drawn from a normal population with an unknown mean JI and a known standard deviation 3. The maximum-likelihood estimate of JI is the mean of the sample observations:

x = 2.7.
We wish to determine upper and lower limits which are rather certain to contain the true unknown parameter value between them. In general, for samples of size 4 from the given distribution the quantity
z=X __ JI ~

will be normally distributed with mean 0 and unit variance. mean, and; is (J/J~. Thus the quantity Z has a density
fz(z) = (z)
= ;_

X is the sample

e -i z2 ,

y2n which is independent of the true value of the unknown parameter; so we can compute the probability that Z will be between any two arbitrarily chosen numbers. Thus, for example,
P[ -1.96 < Z < 1.96] =

1.96

(z) dz = .95.

(1)

-1.96

In this relation the inequality - 1.96 < Z, or -1.96 < is equivalent to the inequality
JI <X
X - JI

+ ;(1.96) =

+ 2.94,

and the inequality Z < 1.96

CONFIDENCE INTERVALS

375

is equivalent to
J1

> X - 2.94.

We may therefore rewrite Eq. (1) in the form

P[X - 2.94

< J1 < X + 2.94]

.95,

(2)

and substituting 2.7 for X we obtain the interval ( - .24, 5.64).

It is at this point that a certain abuse of language takes place since the random interval (X - 2.94, X + 2.94) and the interval (- .24, 5.64) are each called a confidence interval, or more precisely a 95 percent confidence interval. [The interval (- .24,5.64) is the value of the random interval (X - 2.94, X + 2.94) when X = 2.7.] The meaning of Eq. (2) is the following: The probability that the random interval (X - 2.94, X + 2.94) covers the unknown true mean J1 is .95. That is, if samples of size 4 were repeatedly drawn from the normal population and if the random interval (X - 2.94, X + 2.94) were computed for each sample, then the relative frequency of those intervals that contain the true unknown mean J1 would approach 95 percent. We therefore have considerable confidence that the observed interval, here (- .24, 5.64), covers the true mean. The measure of our confidence is .95 because before the sample was drawn .95 was the probability that the interval that we were going to construct would cover the true mean. .95 is called the confidence coefficient. Similarly, intervals with any desired degree of confidence between 0 and 1 can be obtained. Thus, since
,\

P[-2.58 <Z < 2.58] = .99,

a 99 percent confidence interval for the true mean is obtained by converting the inequalities as before to get
P[X - 3.87 < J1 <X

+ 3.87] = .99

and then substituting 2.7 for X to get the interval (- 1.17, 6.57). It is to be observed that there are, in fact, many possible intervals with the same probability (with the same confidence coefficient). Thus, for example, since P[ -1.68 < Z < 2.70] = .95, another 95 percent confidence interval for J1 is given by the interval (-1.35, 5.22). This interval is inferior to the One obtained before because its length 6.57 is greater than the length 5.88 of the interval (- .24, 5.64); it gives less precise information about the location of J1. Any two numbers a and b such that

376

pARAMETRIC INTERVAL ESTIMATION

VIII

FIGURE 1

95 percent of the area under (z) lies between a and b will determine a 95 percent confidence interval. Ordinarily one would want the confidence interval to be as short as possible, and it is made so by making a and b as close together as possible because the relation P[a < Z < b] = .95 gives rise to a confidence interval of length ((J/Jn)(b - a). The distance b - a will be minimized for a fixed area when (a) = (b), as is evident on referring to Fig. 1. If the point b is moved a short distance to the left, the point a will need to be moved a lesser distance to the left in order to keep the area the same; this operation decreases the length of the interval and will continue to do so as long as (b) < (a). Since (z) is symmetrical about z = 0 in the present example, the minimum value of b - a for a fixed area occurs when b = - a. Thus for x = 2.7, (- .24, 5.64) gives the shortest 95 percent confidence interval, and (-1.17, 6.57) gives the shortest 99 percent confidence interval for 1'. In most problems it is not possible to construct confidence intervals which are shortest for a given confidence coefficient. In these cases one may wish to find a confidence interval which has the shortest expected length or is such that the probability that the confidence interval covers a value 1'* is minimized, where
1'* =f:. 1'.

The method of finding a confidence interval that has been illustrated in the example above is a general method. The method entails finding, if possible, a function (the quantity Z above) of the sample and the parameter to be estimated which has a distribution independent of the parameter and any other parameters. Then any probability statement of the form P[a < Z < b] =}' for known a and b, where Z is the function, will give rise to a probability statement about the parameter that we hope can be rewritten to give a confidence interval. This method, Or technique, is fully described in Subsec. 2.3 below. This technique is applicable in many important problems, but in others it is not because in these others it is either impossible to find functions of the desired fOrm or it is impossible to rewrite the derived probability statements. These latter problems can be dealt with by a more general technique to be described in Sec. 4. The idea of interval estimation can be extended to include simultaneous

CONFIDENCE INTERVALS

377

FIGURE 2

estimation of several parameters. Thus the two parameters of the normal distribution may be estimated by some plane region R in the so-called parameter space, that is, the space of all possible combinations of values of Jt and (12. A 95 percent confidence region is a region constructible from the sample such that if samples were repeatedly drawn and a region constructed for each sample, 95 percent of those regions in a long-term relative-frequency sense would include the true parameter point (Po, (l5)(see Fig. 2). Confidence intervals and regions provide good illustrations of uncertain inferences. In Eq. (2) the inference is made that the interval - .24 to 5.64 covers the true parameter value, but that statement is not made categorically. A measure, .05, of the uncertainty of the inference is an essential part of the statement.

2.2 Definition of Confidence Interval

In the previous subsection we attempted to give some feel for the concept of a confidence interval by discussing a simple example. In this subsection we define, in general, what a confidence interval is and in the next subsection describe one method of finding confidence intervals. We assume that we have a random sample Xl' ... , Xn from a density f( . ; 0) parameterized by O. Previously, in Chap. VII, we considered point estimates of say 1:(0), some real function of O. Now we look for an interval estimate of 1:(0).

Definition 1 Confidence interval Let Xl' ... , Xn be a random sample from the density f( ; 8). Let Tl = 11 (Xl' ... , X,.) and T2 = 1 2 (Xb ... , Xn) be two statistics satisfying Tl < T2 for which P8[T1 < 1:(8) < T 2 ] = y, where y does not depend on 8; then the random interval (Tl' T 2 ) is called a 100y percent conjidence interval for 1:(0); Y is called the conjidence coefficient; and Tl and T2 are called the lower and upper confidence limits, respectively, for r(O). A value (tl' t 2 ) of the random interval (Tl' T 2 ) is also called a
100y percent confidence interval for r(O).

IIII

378

PAIt.AMBTRlC INTERVAL ESTIMATION

vm

We note that one or the other, but not both, of the two statistics tl(Xb " ' , Xn) and t 2 (Xl , .,., Xn) may be constant; that is, One of the two end points of the random interval (Tl , T 2 ) may be constant.
One..sided confidence interval Let Xl' ... , X,. be a random sample from the density f( . ; 0). Let Tl = tl(Xl , ... , XJ be a statistic for which PO[Tl < 1:(0)] = y; then Tl is called a one-sided lower corifidence limit for 1:(0). Similarly, let T2 = t iXl , ... , Xn) be a statistic for which Poh(O) < T2] == y; then T2 is called a one-sided upper confidence limit for 1:(0). (y does not depend On 0.) IIII

Definition 2

EXAMPLE 1 Let Xl' ... , Xn be a random sample from f(x; 0)

= q,O.9(X).

Set Tl = tl(Xl , , .. , XJ = X - 6/~ and T2 = t 2(Xl , .. ,' XJ = X + 61 ~; then (Tl' T2 ) constitutes a random interval and is a confidence interval for 1:(6)

= Owithconfidencecoefficienty =

Po[X -

61J-;i < 0 < X + 61J25)

61j;i]=Po[-2
.9544.

< (X - 6)/(3/~) <2]= <1>(2) - <1>(-2) =.9772 -.0228 = Also, if a random sample of 25 observations has a sample mean is also

of, say, 17.5, then the interval (17.5 - 61J25, 17.5 + called a 95.44 percent confidence interval for O.

'1111

Remark If a confidence interval for () has been determined, then, in

essence, a whole family of confidence intervals has been determined. More specifically, for a given lOGy percent confidence interval estimator of a 100y percent confidence-interval estimator of 1:(0) can be obtained, where 1:(') is any strictly monotone function. For example if 1:(') is a monotone, increasing function and (Tl = tl(Xl , ... , XJ, T2 = t 2 (Xl , , Xn is a 100y percent confidence interval for 0, then (1:(Tl ), 1:(T2 is a 100y percent confidence interval for 1:(0) since

As was the case in point estimation, our problem is twofold: First, we need methods of finding a confidence interval, and, second, we need criteria fOl~ comparing competing confidence intervals or for assessing the goodness of a confidence interval. In the next subsection, we will describe One method of finding confidence intervals and call it the pivotal-quantity method.

CONFIDENCE INTER VALS

379

FIGURE 3

2.3

Pivotal Quantity

As before, we assume a random sample Xl' .. , X,. from some density f('; 0) parameterized by O. Our object is to find a confidence~interval estimate of'r(O), a real-valued function of O. 0 itself may be vector-valued.

Definition 3 Pivotal quantity Let Xl, ... , X,. be a random sample from the density f( ; 0). Let Q = ?(Xl , ... , X,.; 0); that is, let Q be a function of Xl' .. , X,. and O. If Q has a distribution that does not depend on 0, then Q is defined to be a pivotal quantity. IIII

EXAMPLE 2 Let Xv ... , X,. be a random sample from f(x; 0) = <P8.9(X), X - 0 is a pivotal quantity since X - 0 is normally distributed with mean 0 and variance 91n. Also (X - O)/(3/~) has a standard normal distribution and, hence, is a pivotal quantity. On the other hand, XIO is not a pivotal quantity since X10 is normally distributed with mean unity and variance 910 2 n, which depends on O. IIII Our hope is to utilize a pivotal quantity to obtain a confidence interval.

Pivotal-quantity method If Q = ?(Xv ... , X,.; 0) is a pivotal quantity and has a probability density function, then for any fixed 0 < y < 1 there will exist ql and q2 depending on y such that P[ql < Q < q2] = y. Now, if for each possible sample value (Xh . , X,.), ql < ?(x l , ... ,X,.; 0) < q 2 if and only if tl (Xl' ... ,xJ < 'reO) < t 2(Xl' ... , X,.) for functions tl and t 2 (not depending on 0), then (Tl' T 2) is a 100y percent confidence interval for 'reO), where Ti = ti(Xb , X,.), i = 1, 2.
Before illustrating the pivotal-quantity method with a simple example we make several comments. First, ql and q 2 are independent of 0 since the distribution of Q is. Second, for any fixed y there are many possible pairs of numbers ql and q2 that can be selected so that P[ql < Q < q2] = y. See Fig. 3. Different pairs of ql and q2 will produce different tl and t 2' We should want to select that pair of ql and q2 that will make t 1 and t 2 close together in some sense. For instance, if t 2(Xl , ... , X,.) - tl (Xl! ... , X,.), which is the length of the confidence

380

PAR.AMETRIC INTEllVAL ESTIMATION

vm

FIGURE 4

interval, is not random, then we might select that pair of ql and q2 that makes the length of the interval smallest; or if the length of the confidence interval is random, then we might select that pair of ql and q2 that makes the average length of the interval smallest. As a third and final comment, note that the essential feature of the pivotalquantity method is that the inequality {ql < 9(Xl, ... , Xn; 8) < q2} can be rewritten, or inverted or" pivoted," as {tl(X l , ... , xn) < -r(8) < tix l , ... , xn)} for any possible sample value Xl' ... , x n [This last comment indicates that "pivotal quantity" may be a misnomer since according to our definition Q = 9(Xl , ... , Xn; 8) may be a pivotal quantity, yet it may be impossible to "pivot" it.]

EXAMPLE 3 Let Xl' ... , Xn be a random sample from o, l(X), Consider estimating -r(8) = 8. Q = 9(Xl , ... , Xn; 8) = (X - 8)/(JlTn) has a standard normal distribution and, hence, is a pivotal quantity. fQ(q) = (q). For given I' there exist ql and q2 such that P[ql < Q < q2] = I' (in fact, there exist many such ql and q2)' See Fig. 4. Now {ql < (x - e)IJ1/n < q2} if and only if {x - q2Jl/n < e < x -qlJl/n}; so (X - q2JlTn, X .:.. qlJl/n) is a 1001' percent confidence interval for e. The length of the confidence interval is given by (X - qlJ lin) - (X - q2#) = (q2 - ql)JlTn; so the length will be made smallest by selecting ql and q2 so that q2 - ql is a minimum under the restriction that I' = P[ql < Q < q2] = <I>(q2) - <I>(ql), and q2 - ql will be a minimum if ql = -q2' as can be seen from Fig. 4. IIII

The steps in the pivotal-quantity method of finding a confidence interval are two: First, find a pivotal quantity, and, second, invert it. We will comment further on techniques for finding pivotal quantities in Sec. 4. The method is thoroughly exploited in the next section.

SAMPLING FROM THE NORMAL DISTRIBUTION

381

SAMPLING FROM THE NORMAL DISTRIBUTION

Let Xl' ... , X It be a random sample from the normal distribution with mean Il and variance 0'2. The first three subsections of this section are generated by. the cases (i) confidence interval for Il only, (ii) confidence interval for 0'2 only, and (iii) simultaneous confidence interval for Il and 0'. The fourth subsection considers a confidence interval for the difference between two means.

3.1

Confidence Interval for the Mean

There are really two cases to consider depending on whether or not 0'2 is known. We leave the case 0'2 known as an exercise. (The technique is given in Example 3.) We want a confidence interval for Il when 0'2 is unknown. In our general discussion in Sec. 2 above OUr parameter was denoted by O. Here 0 = (Il, 0'), and -r(0) = Il. We need a pivotal quantity. (X -Il)/(O'/,/;;) has a standard normal distribution; so it is a pivotal quantity, but {ql < (x -Il)/(O'/Jn) < q2} cannot be inverted to give {t 1 (Xl' ... , x,.) < Il < t 2(Xl' ... , x,.)} for any statistics tl and t 2 . The problem with (X - Il)/(O'j);;) seems to be the presence of 0'. We look for a pivotal quantity involving only Il. We know that

has a t distribution with n - I degrees of freedom. [Recall that 8 2 = (Xi - X)2j(n - 1).] So (X - ll)j(8j);;) has a density that is independent of Il and 0'2; hence it is a pivotal quantity. Now one has {ql < (x - Il)/(<J/Jn) < q2} if and only if {x - q2(<JjJ;;) < Il < x - ql(<J/);;)}, where ql and q2 are such that P[ql < (X - 1l)J(8/J;;) < q2] = 1'; therefore, (X - q2(8/.jn), X - ql(8/Jn is a lOOy percent confidence interval for Il. The length of this confidence interval is (q2 - ql)(8/Jn), which is random~ For any given sample the length will be minimized if ql and q2 are selected so that q2 - ql is a minimum. A little reflection will convince One that ql and q2 should be symmetrically selected about 0, Or the following argument can be advanced. We seek to minimize

subject to
(3)

382

pARAMETRIC INTERVAL ESTIMATION

vm

where fr(t) is the density of the t distribution with n - 1 degrees of freedom. Equation (3) gives q2 as a function of ql' and differentiating Eq. (3) with respect to ql yields

To minimize L, we set dLjdql

0; that is,

but

J~

8 (d q 2

) dql - 1

= In

8 (frlql)

) frlq2) - 1

=0

S:!

if and only if fT(ql) = fT(q2)' which implies that ql = q2 [in which case fT(t) dt =f:. y] or ql = -q2' ql = -q2 is the desired solution, and such ql and q2 can be readily obtained from a table of the t distribution.

3.2

Confidence Interval for the Variance

Again there are two cases depending On whether or not Il is assumed known, and again we leave the case Il known as an exercise. We want a confidence interval for (1'2 when Il is unknown. We need a pivotal quantity that can be inverted. We know that Q = L (Xl - X)2 = (n - 1)82
(1'2 (1'2

has a chi-square distribution with n - 1 degrees offreedom; hence Q is a pivotal quantity. Also, one has

if and only if

so
(n - 1)8 (
2 ,

(n - 1)8

q2

ql

SAMPLING FROM THE NORMAL DISTRIBUTION

383

Chi-square density with n - 1 degrees of freedom

FIGURE 5

is a looy percent confidence interval for (12, where ql and q2 are given by P[ql < Q < q2] = y. See Fig. 5. ql and q2 are often selected so that P[Q < ql] = P[Q > q2] = (1 - y)/2. Such a confidence interval is sometimes referred to as the equal-tails confidence interval for (12. ql and q2 can be obtained from a table of the chi-square distribution. Again, we might be interested in selecting ql and q2 so as to minimize the length, say L, of the confidence interval.

L = (n - 1)82 - - -

(1 I ) . ql q2

Letfa(q) be a chi-square density with n - 1 degrees of freedom; then differen-

tiating

with respect to ql yields

and so

dL
dql

.=

(n _ 1)8 2(- 12 ql

q 12 d 2) = (n _ 1)8 2(- 12 q2 dql ql

+ ~fa(ql)) = 0,
q2fa(q2)

which implies that q;fa(ql) = q~fa(q2)' The length of the confidence interval will be minimized if ql and q2 are selected so that

subject to

A solution for ql and q2 can be obtained by trial and error or numerical integration.

384

pAJt.AMETRIc INTBllVAL ESTIMATION

vm

FIGURE 6

j -

t. 97s J111/n

t'97S

J {]l/n

We might note that for any ql and q2 satisfying

is a 10Dy percent confidence interval for

(1.

3.3

Simultaneous Confidence Region for the Mean and Variance

In constructing a region for the joint estimation of the mean /1 and variance (12 of a normal distribution, one might at first be inclined to use Subsec. 3.1 and 3.2 above. That is, for example, one might construct a confidence region as in Fig. 6 by using the two relations

and
2 2 en 1)8 (n - 1)8 ] 2 p [ 2 < (1 < 2 X.975 X.025

.95,

(5)

where t.975 is the .975th quantile point of the t distribution with n - 1 degrees of freedom and X~25 and X.~75 are the .025th quantile point and .975th quantile point, respectively, of the chi-square distribution with n - 1 degrees of freedom. The region displayed in Fig. 6 does indeed give a confidence region for (/1, (12), but we do not know what its corresponding confidence coefficient is. [It is not .95 2 since the two events given in Eqs. (4) and (5) are not independent.] A confidence region, whose corresponding confidence coefficient can be

SAMPLING FROM 11IE NORMAL DISTRmunoN

385

2
---(1

=
2

(n - l)a2 "
ql

(Il - X)I =

ql(1

FIGURE 7

L-------~~~---------+Il

readily evaluated, may be set up, however, using the independence of X and S2. Since and are each pivotal quantities, we may find numbers ql' q2' and qi such that
p [ -q, <

~/ln <q,]

= y,

(6)

and (7) Also, since Ql and Q2 are independent, we have the joint probability
P [ -ql <

X-

(1/ J n

It

(n - 1)S ] < ql; q2 < 2 < q2

(1

= 1'11'2'

(8)

The four inequalities in Eq. (8) determine a region in the parameter space, which is easily found by plotting its boundaries. One merely replaces the inequality signs by equality signs and plots each of the four resulting equations as functions of It and (12. A region such as the shaded area in Fig, 7 will result. We might note that a confidence regiOn for (It, (1) could be obtained in exactly the same way; the equations would be plotted as fUnctions of (1 instead of (12, and the parabola in Fig. 7 would become a pair of straight lines given by
p.

x + ql(1/J;; intersecting at x On the It axis.

The region that we have constructed does not have minimum area for fixed 1'1 and 1'2' Its advantage is that it is easily constructible from existing tables and it will differ but little from the region of minimum area unless the sample size is smalL The region of minimum area is roughly elliptic in shape and difficult to construct.

386

pARAMETRIC INTERVAL ESTIMATION

vm

3.4 Confidence Interval for Difference in Means

Let Xl' ... , X m be a random sample of size m from a normal distribution with mean 1'1 and variance (f2, and let Yb . , Yn be a random sample of size n from a normal distribution with mean 112 and variance (f2. Assume that the two samples are independent of each other. We want a confidence interval for 112 - Ill' Y - X is normally distributed with mean 112 - 1'1 and variance (f2jn + (fIjm. L (Xi - X)2j(f2 is chi-sQuare-distributed with m - 1 degrees of freedom, and L (Yi - Y)2j(f2 is chi-sQuare-distributed with n - 1 degrees of freedom; hence

L (Xi (12

X)2 + L (Yi - Y)2

(12

is chi-sQuare-distributed with m + n - 2 degrees of freedom.

Finally,

Q=

[( y -

X) - (1'2 - IlI)]jJ~(12-jm

J[L (Xi - X)2 J(ljm

+L

(Yi X)2

+ (12jn Y)2]j(12(m + n +L
(Yi -

2)

( Y - X) - (1l2 - Ill)

+ 1jn) [2: (Xi -

Y)2]j(m

+' n - 2)

( Y - X) - (1l2 - Ill)
J(ljm

+Ijn)(S;)

has a t distribution with m + n - 2 degrees of freedom. Thus it follows that y = P[ - t(1 +y)/2 < Q < t(1 +Y)/2], where t(l +y)/2 is the [(1 + y)/2]th quantile point of the t distribution with m + n - 2 degrees of freedom. is an unbiased estimator of the common variance (f2. (The subscript p can be thought of as an abbreviation for "pooled"; is a pooled estimator of (12, the two samples being pooled together.) Now

S;

S;

(y - x) - (1l2 - Ill)
-t(1+Y)/2<

J (l/m + l/n)o;
< (y - x)

<t(1+y)/2

if and only if

(,V - X) - t (1+1){2
hence

(:

+~) d~ <

/12 -

/11

+ +rH.J (~ +~)d!;
t(1

(- -

(Y-X)-t(1+y)/2

J(1 1) 2 - -

m+n Sp,(Y-X)+t(1+Y)/2

J(l 1)S2)
m+~
p

(9)

is a lOOy percent confidence interval for 112 - Ill'

METHODS OF FINDING CONFIDENCE INTERVALS

387

We assumed above that we had two independent random samples. Now assume that (Xl' Yl ),, (X,., Y,.) is a random sample from the bivariate normal distribution with parameters given by III = 4'[X], Ilz = 4'[ Y], O'I = var[X), O'~ = var[ y], and p = cov [X, Y]/ 0'10' 2 The 0 bject is to find a confidenceinterval estimate of J.lz - Ill' Let Die = Y t - Xb i = 1, ... , n; then Db ... , D,. are independent and identically distributed random variables with Common normal distribution having mean J.lD = Ilz - J.ll and variance O'~ = O'i + O'i 2pO'I 0' 2 Pretending that Db ... , D,. is our random sample and proceeding as in Subsec. 3.1, we obtain the following 100y percent confidence interval for
112 - Ill:

(-

D-

t(1+'1)/2

Jl:

(D, - D)2 D, n(n - 1)

+ t(I+'1)/2

(D, - D)2) n(n - 1)

(10)

where t(1 +'1)1 2 is the [(1 + y)/2]th quantile point of the t distribution with n - 1 degrees of freedom. The above obtained confidence interval for 112 - III is often referred to as the confidence interval for the difference in means for paired observations. The ith X observation is paired with the ith Yobservation.

METHODS OF FINDING CONFIDENCE INTERVALS

In this section we will discuss two methods of obtaining a confidence interval. A third method will be presented in Chap. IX .

./ 4.1

Pivotal-quantity Method

We described the pivotal-quantity method of finding confidence intervals in Sub sec. 2.3, but we left unanswered the question of whether or not a pivotal quantity would actually exist for a given problem. The following remark gives a partial answer to this question.

Remark If Xb ... , Xn is a random sample from f( ; 8), for which the

corresponding cumulative distribution fUnction F(x; 8) is continuous in x, then, by the probability integral transform, F(Xi; 8) has a uniform distribution over the interval (0, 1). Hence -log F(Xi ; 8) has the density e-ul(o,oo)(u) since P[-logF(Xi; 8);:::u] =P[logF(Xj ; 8)< -u]= U U P[F(X,; 8) se- ] = e- for u > O. Finally log F(X. 8) has a gamma distribution with parameters n and I; that is, "

388

pARAMETRIC INTERVAL ESTIMATION

VIII

p[ -log
=

q2 < - ,t,iogF(X,; 9) < -log q,]

-IOgq. -logqz

__

zit

le- z

dz
(11)

r(n)

p[q, < lJ.F(X,; 9) < q2]

TI F(Xi ; 8),
It

So or i= 1

L log F(X
i"'" 1

j ;

8),

is a pivotal quantity.

II/I

The remark shows that any tim~ that we sample from a population having a continuous cumulative distribution function, a pivotal quantity exists. Note that as of yet we have no assurance that the pivotal quantity exhibited by the remark can be utilized to find a confidence interval. If, however, F(x; 8) is monotone in 8 for each x, then

TI
i"" 1

F(Xi; 8) is also monotone in 8 for each

Xl' ... , Xn , and such monotonicity allows one to find a confidence interval for 8.

We see from Fig. 8 that ql <

i

TI
1

F(xj; 8) < q2 if and only if 11 (X 1 ,

"',

xn) < 8 <

1 2(Xl, ... , x n), where II and 12 are defined as indicated.

EXAMPLE 4

Let Xl, ... , Xn be a random sample from the density r(x; 8) = 8xO- 1I(o.1)(x); then F(x; 8) = XOI(O.llx) + I[t,oo)(x), If ql and q2 are selected [see Eq. (11)] so that
l'

=
=

p[q, < laF(X,; 9) < q2]

p[q, < ,Ij,xr < q2] q, < 9 log Iv'XI < log q2]
i= 1

= P[IOg

= P[-log q2 < -810g

=P
log q 2
It

IT Xi< -log ql]

]
' It

<8<

log q 1 10gJ:\ Xi

[ 10glU Xi

METHODS OF FINDING CONFIDENCE INTERVALS

389

1~----------------------~--

FIGURE 8

then

is a 100)' percent confidence interval for 8.

IIII

We conclude this subsection with two further comments regarding the existence of pivotal quantities. First, if 8 is a location parameter, then Xi - 8 has a distribution independent of 8 by definition and, hence, is a pivotal quantity as are a variety of other random quantities, including L Xi - n8, Yj - 8, Yl + Yn - 28, etc. Second, if 8 is a scale parameter, then by definition Xd8 is distributed independently of 8 and, hence, is a pivotal quantity as are L Xd8, YJl8, etc.

4.2

Statistical Method
t

As usual, we assume that we have a random sample Xl' ... , Xn from the density f(' ; ( 0), We further assume that the parameter 80 is real and that the paramis some interval. (In this SUbsection, we will let 80 denote the eter space true parameter value.) We seek an interval estimate of 80 itself. Let T = t(Xb ... , Xn) be some statistic. The statistic T can be select~d in several ways. For instance, if a sufficient statistic (unidimensional) exists, then T could be taken to be a sufficient statistic; or if no sufficient statistic exists, Tcould be taken to be a point estimator, possibly the maximum-likelihood estimator, of 80 , The actual choice of T might depend on the ease with which the operations that need to be performed to obtain the confidence interval can be performed. One of those operations will be the determination of the density of T.

390

pAJlAMBTRlc INTERVAL ESTIMATION

VBI

~
Area
h1 (6)

= Pl
t

FIGURE 9

Let fT(t; 0) denote the density of T. We will proceed as though T is a continuous random variable; although the technique will also work for T as a discrete random variable. We can define two fUnctions) say hl(O) and h2 (0)) as follows: and
(12)

where Pl and P2 are two fixed numbers satisfying 0 < Pl) 0 < P2 ) and Pl + P2 < 1. See Fig. 9. hl(O) and h2 (0) can be plotted as functions ofO. We will assume that both hl ( .) and h2 () are strictly monotone) and for our sketch we will assume that they are monotone) increasing functions. We know that hl (0) < h2 (0). See Fig. 10. Let to denote an observed value of T; that is, to = t(Xl, ... , xn) for an observed random sample Xb .. , Xn Plot the value of to on the vertical axis in Fig. 10, and then find Vl and V 2 as indicated. For any possible value of to, a corresponding Vl and V2 can be obtained, so Vl and V2 are functions of to; denote these by Vl = Vl(tO) and V2 = V2(t O )' The interval (Vb V 2) will turn out to be a 100(1 - Pl - P2) percent confidence interval for 00 , To argue that this is so, let us repeat Fig. 10 as Fig. 11 and add to it. (Figure 10 indicates the method of finding the confidence interval.)
t

to

---------

FIGURE 10

METHODS OF FINDING CONFIDENCE INTERVALS

391

FIGURE 11

We see from Fig. 11 that hi (0 0) < to = t(X I , ... ~ Xn) < h 2(00) if and only if VI = VI(Xb ~ xn) < 00 < V2 = V2(XI~ ... ~ xn) for any possible observed sample (Xj ~ ... ~ x n). But by definition of hi (.) and h2(')~
P8o[h l (0 0) < t(Xb ... ~ Xn) < h 2(00)]
=

1 - PI - P2;

so
P80 [VI(Xh ' ~ Xn) < 00 < V2(Xh
...

~ Xn)]

= 1 - PI - P2;

that is~ as stated~ (VI~ V 2) is a 100(1 - PI - P2) percent confidence interval for 00 ~ where Vi = Vl(X b ... , Xn) for i = 1, 2. We might note that the above procedure would work even if hi (.) and h2 ( .) were not monotone fUnctions, only then we would obtain a confidence region (often in the form of a set of intervals) instead of a confidence interval.

EXAMPLE 5 Let Xl' ... , Xn be a random sample from the density f(x; ( 0) = (1/(Jo)I(0.8o)(x), We want a confidence interval for 00 , Y n = max [Xl, ... ~ Xn] is known to be a sufficient statistic; it is also the maximumlikelihood estimator of(Jo' We will use Yn as our statistic Tthat appears in the above discussion; then
fT(t; 8) = n (~r-l

~ I(0.6)(t).

For given PI and P2 , find hi (0) and h2(0). PI = S~1(8) ntn-1o- n dt implies that Ji1 (8) t n- l dt = onpl/n, which in tum implies [hl(o)]n/n = onpl/n~ or finally hi CO) = Opt/no Similarly, P2 = S:1(8) ntn-Io- n dt implies that 8" - [h 2(o)]n = 8"P2 or h 2(0) = 0(1 - P2)I/n. See Fig. 12, which is Fig. 10 for the example at hand.

392

PAIlAMETlUC INTERVAL ESTIMATION

VID

FIGURE 12

-=~----------------~----6

Vl

For observed to = max [Xl' ,Xn ], VI is such that h 2 (Vl) = to; that is, hlv 1) = Vl(1 - P2)I/n = to or VI = to(l - P2)-lln. Similarly, V 2 = top-lIn. So a 100(1 - PI - pz) percent confidence interval for 00 is given by (Y,.(1 - P2)-I/n, Y nPl l !n). We could worry about selecting PI and Pl so that the confidence interval is shortest subject to the restriction that 1 - PI - Pl = y. The length of the confidence interval is
L

= Y~[pilin -

(1 - P2) -lIn]

and so the length will be shortest if PI and Pl are picked so as to minimize pill,. - (1 - P2)-1/1I subject to 1 - PI - P2 = Y and 0 <PI + Pl < 1, which is accomplished by picking P2 = 0 and PI = 1 - y. We might note that Yn/O is a pivotal quantity and a confidence interval for 0 can be obtained more easily using the pivotal-quantity method. / // / We observe in the example above, and in general for that matter, that hI (0) and hiO) are really not needed. For a given observed value to = I(Xb , xn) of the statistic T, we need to find VI = 'Vt(Xb ... , xJ) and V 1 = 'Vl(Xb , x 1I ). V2 can be found by solving for 0 in the equation \ Pt
V1

J f

hl(tJ)=to

\\

/,.(t; 8) dt;
\

(13)

-00

is the solution;

VI

can be found by solving for 0 in the equation

00

' (14)

Pl =
VI

fT(t; 8) dt;

h;z(6) =to

is the solution. We mentioned at the outset that the method would work for discrete random variables as well as for continuous random variables. Then the integrals in Eqs. (12) to (14) would need to be replaced by summations. Two

LARGE-SAMPLE CONFIDENCE INTERVALS

393

popular discrete density functions are the Bernoulli and Poisson. One could be interested in confidence-interval estimates of the parameters in each. In Example 6 to follow we will consider the Bernoulli density function; the Poisson case is left as an exerCIse.

EXAMPLE 6 Let Xh . , Xn be a random sample from the Bernoulli density; that is, P[X = 1] = 00 = 1 - P[X = 0]. We know that T = L Xi is a sufficient statistic; furthermore, T has a binomial distribution; that is,
P[T =

t1 = (~)8W - (}o)"-' for t= 0, 1, ... , n.

We want a confidence-

interval estimate for 00 , Suppose we observe T = to (necessarily an integer). According to Eqs. (13) and (14) we need to solve for 0 in each of the equations

and

To actually solve these equations, a table of the binomial distribution is useful. If Pt .0509, P2 = .0159, n = 20, and if T = 4 is observed then the 93.33 percent confidence interval (.05, .40) for 00 is obtained. 11I1

LARGE-SAMPLE CONFIDENCE INTERVALS

We have seen in our studies of point estimation that it is sometimes possible to find a sequence of estimators, say Tn = tn(Xt , .. , Xn), of 0 in a density f(' ; 0) that are asymptotically normally distributed about 0; that is, Tn is approximately normally distributed with mean 0 and variance, say, 0';(0), where 0';(0) indicates that the variance is a function of 0 (since it will ordinarily depend on 0) and the sample size n. In particular, we have seen in Sec. 9 of Chap. VII that for large samples the maximum-likelihood estimator, say Gn = 8n(Xt , .. , X n ), for a parameter 0 in a density f(; 0) is approximately normally distributed about 0 under rather general conditions. The large-sample variance of the maximumlikelihood estimator was seen to be, say, 0' 2(0) ..

1 -1 _ 2 2 n10 [{(0/00) log f (X; O)} ] - -nS'":""o-:":[ (--:a2;-1a::-::O-::')-1 - o-g-f-(x-; 0:::-:-::)]'

(15)

394

p.A.llAMln"l';UC INTBIlVAL ESTIMATION

VIII

When such a sequence of asymptotically normally distributed estimators {Tn} of o exists, it is sometimes possible to obtain approximate confidence intervals quite easily. (Tn - 6)/0',.(0) can be treated as an approximate pivotal quantity, and, therefore, for large sample size a confidence interval with an approximate confidence coefficient y may be determined by converting the inequalities
T -0 -z < ;n(O) < z,
(16)

where z = Z(1 +1)/2 is defined by fl)(z(l +y)/2) = (1 + y)/2 or fl)(z) - fl)( -z) = y. The above described method will always work to find a large-sample confidence interval provided the inequality -z < (Tn - O)IO'n(O) < z can be inverted.

EXAMPLE 7 Let Xb ... , Xn be a random sample from the density f(x; 0) = Oe-ox/(o. )(x). We know (see Example 51 in Chap. VII) that the maximum-likelihood estimator of 0, which is 1/Xn , has an asymptotic normal distribution with mean 0 and variance equal to
m9'9[{(81 80) 10gf(X;
1 ----------::- -

-.
n

Therefore,
y,;:;;p [ -z<

1/Xn - 0 ] fii2i:. <z V 021n

-ZO 1 ZO] =p [ v1n<Xn- O <vIn

=p [

l/Xn < 1 + zlvln

II

<

l(Xn ] , 1 - ~/Jn
\
\

and hence

is a large-sample confidence interval for 0 with an approximate confidence coefficient y, where z is given by fl)(z) - fl)( -z) = y. 1111

EXAMPLE 8 Consider sampling from the Bernoulli distribution with parameter 0 = P[X = 1] = 1 - P[X = 0]. The maximum-likelihood estimator

LARGE-8AMPLE CONFIDENCE INTERVALS

395

of 0 is Q = X n, and it has variance u;(O) = 0(1 - O)ln. An approximate 100y percent confidence interval for 0 is obtained by converting the inequalities in
p [ -z<

0-0
J 0(1 - O)ln

<z] :::::::y

to get
p[2nQ + Z2 - zJ4nQ + Z2 - 4nQ2 2(n + Z2)

<

(J

<

2nQ + Z2 + z)4nQ + Z2 - 4nQ2] "" 2(n + Z2) "" Y .

(17)

These expressions for the limits may be simplified since in deriving the large-sample distribution certain terms containing the factor l/.j;i are neglected; that is, the asymptotic normal distribution is correct only to within error terms of size a constant times II);;. We may therefore neglect terms of this order in the limits in Eq. (17) without appreciably affecting the accuracy of the approximation. This means simply that we may omit all the Z2 terms in Eq. (17) because they always occur added to a term with factor n and will be negligible relative to n when n is large to within the degree of approximation that we are assuming. Thus Eq. (17) may be rewritten as
p

- - )(;)(1 [e n
Z

-Q)

~ + <0<e

J0(l -(;]
n
n

:::::::

y.

(18)

In particular,
P E> - 1.96~

[-

1~(1 n-Q) < (J <

1.96

JQ(l - Q)]

: : : : .95

gives an approximate 95 percent confidence interval for 0 for large samples. IIII We may observe that Eq. (18) is just the expression that would have been obtained had Q been substituted for 0 in u;(O). The substitution would imply that

A- 0 )~(l - (;)/n

396

PAIlAMETRIC INTERVAL ESTIMATION

VIII

is approximately normally distributed with mean 0 and unit variance. It is, in fact, true in general that in the asymptotic normal distribution of a maximumlikelihood estimator ~ the variance (j;(0) may be replaced by its estimator (j;(~) without appreciably affecting the accuracy of the approximation. We shall not prove this fact but shall use it because it greatly simplifies the conversion of the inequalities that is required to get the large-sample confidence intervals. For instance,

readily converts to give

P[~ - z(jn(~)

< 0 < ~ + z(jn(~)] ~ y,

(19)

where ~ is the asymptotically normally distributed maximum-likelihood estimator of 0, (jn(~) in this expression is the maximum-likelihood estimator of (jn(O) (which is the large-sample standard deviation of ~), and z is given by <I>(z) - <1>( - z) = y. We noted in Sec. 9 of Chap. VII that under regularity conditions the joint distribution of the maximum-likelihood estimators of the components of a k-dimensional parameter is asymptotically normally distributed. Although we will not so argue, such a result could be used to obtain a large-sample confidence regIOn. The large-sample confidence intervals presented in this section have an optimum property which we shall point out but not prove. Recall that in the earlier sections, particularly Sec. 3, of this chapter we were concerned with finding the shortest interval for a given probability. Loosely speaking, an ana10gous optimum property of large-sample confidence intervals based on maximumlikelihood estimators is this: Large-sample confidence intervals based on the maximum-likelihood estimator will be shorter, on the average, than intervals determined by any other estimator.

BAYESIAN INTERVAL ESTIMATES

In Sec. 7 of Chap. VII we examined what is called Bayes estimation. There we assumed that a random sample, say Xl' ... , X n , from some density f(' ; 0) = f('IO) was available, where the form of the functionf( '1') was known and the fixed value of 0 was unknown. We further assumed that the unknown fixed value of 0 was the value of a random variable e with known density, denoted by ge(') and called the prior density of e. We then used this additional knowledge

BAYESIAN INTERVAL ESTIMATES

397

of a known prior density to define the posterior distribution of , and from this posterior distribution we defined the posterior Bayes point estimator of O. In this section we use this same posterior distribution of to arrive at an inter1(ai estimator of O. If 1(' 10) is the density sampled from and ge(') is the prior density of , then the posterior density of given (Xl' ... , Xn) = (Xl'" . , xn) is [recall Eq. (19) of Chap. VII]

f",x, =xo ... x.= elll x" ... , xJ =

[Qf(x. 1 0) ]9,,(11) J[Qfexd 0)]9,,(0) dO:

(20)

For fixed /', any interval,. say (tl' t 2), satisfying

is defined to be a l00/, percent Bayesian interval estimate of O. In practice, one would naturally pick those tl and t2 satisfying Eq. (21) for which t2 - tl is smallest. Note that ti = ti~Xb .. , X,.); that is, t{ is some function of the observations
Xi) , X n

EXAMPLE 9 Let Xl' ... , Xn be a random sample from the normal density with mean 0 and variance 1. Assume that has a normal density with mean Xo and variance 1. Consider estimating O. We saw in Example 44 of Chap. vn that the posterior distribution of is normal with mean

L xi/en + 1) and variance lj(n + 1). o

/' =
=

We seek tl and t2 satisfying

xn) dO

f fejx .... xI. ... ,xn ... xJO I

t2-

Xh ., .,

t1

(t2 - 'tXi/(n +

1) _(t l then

'tXi/(n +

1).

Jl/(n + 1)
If z is such that (z) - ( - z)
= /"
n

Jl/(n + 1)

(22)

t2

~ Xi
n+l

+z

J--1---n+l

and

398

PARAMETRIC INTERVAL ESTIMATION

VIII

gives the shortest 1001' percent Bayesian interval estimate of Note that the corresponding 1001' percent confidence-interva1 estimate of is given

e.

by

(t xJn - zJlIn, t xJn + zJi/n).

The only difference in the results

of the two methods for this example is that the sample size seems to increase by I and the apparent" additional observation" is the mean of the assumed prior normal distribution. / // /

PROBLEMS
1 Let X be a single observation from the density
I(x; 6)
=

(JxfJ-1I(o.1)(X),

2 3

where (J > O. (a) Find a pivotal quantity, and Use it to find a confidence-interval estimator of (J. (b) Show that (Y/2, Y) is a confidence interval for (J. Find its confidence coefficient. Also, find a better confidence interval for (J. Define Y = -l/log X. Let Xl, "', Xn be a random sample from N(J, 8), (J > O. Give an example of a pivotal quantity, and Use it to obtain a confidence-interval estimator of (J. Suppose that Tl is a 100y percent lower confidence limit for T(J) and T2 is a 100y percent upper confidence limit for T(6). Further assume that PfJ(Tl < T 2] = 1. Find a 100(2y - 1) percent confidence interval for T(8). (Assume y > ~.) Let Xl, ... Xn denote a random sample from I(x; 6) = l(fJ-~. fJ+~)(x). Let Y1 < ... < Y,. be the corresponding ordered sample. Show that (Yl~ Yn) is a confidence interval for (J. Find its confidence coefficient. Let Xl, "', X,. be a random sample from/(x; 8) = (Je-fJxl(o. OO)(x). (a) Find a 100y percent con'fidence interval for the mean of the population. (b) Do the same for the variance of the population. (e) What is the probability that these intervals COver the true mean and true variance, simultaneously? (d) Find a confidence-interval estimator of e - fJ = P(X > I]. (e) Find a pivotal quantity based only on YJ, and use it to find a confidenceinterval estimator of (J. (Yl = min (Xl, . ", X,.].) X is a single observation from (Je-fJxl(o. OO)(x), where (J > O. (a) (X, 2X) is a confidence interval for l/(J. What is its confidence coefficient? (b) Find another confidence interval for 1/(J that has the same coefficient but smaller expected length. Let Xl, X2 denote a random sample of size 2 from N(J, 1). Let Yl < Y 2 be the corresponding ordered sample. (a) Determine yinP(Yl < (J < Y 2 ] = y. Find the expected length oftheinterval
(Y1 , Y 2 ).
(b)

Find that confidence-interval estimator for 8 using X - (J as a pivotal quantity that has a confidence coefficient y, and compare the length with the expected length in part (a).

PROBLEMS

399

Consider random sampling from a normal distribution with mean p. and variance
a2 (a) (b)

Derive a confidence interval estimator of p. when a 2 is known. Derive a confidence interva1 estimator of a 2 when p. is known. 9 Find a 90 percent confidence interval for the mean of a normal distribution with a = 3 given the sample (3.3, -.3, -.6, -.9). What would be the confidence interva1 if a were unknown? 10 The breaking strengths in pounds of five specimens of manila rope of diameter 3 1 6 inch were found to be 660, 460, 540, 580, and 550. (a) Estimate the mean breaking strength by a 95 percent confidence interval assuming normaJity. (b) Estimate the point at which only 5 percent of such specimens would be expected to break. (c) Estimate a 1 by a 90 percent confidence interval; a1so a. (d) Plot an 81 percent confidence region for thejoint estimation of p. and a 1 ; for p. and a. 11 A sample was drawn from each of five populations assumed to be normal with the same variance. The va1ues of (n - 1)~)2 = L (X, - %)2 and n, the sample size, were 82 : 40 m W 42 ~
n:

12

13
14
15

16

17
18

Find 98 percent confidence limits for the common variance. Develop a method for estimating the ratio of variances of two norma1 populations by a confidence interval (1"'What is the probability that the length of a t confidence interva1 for p. when sampling from a norma1 distribution win be less than a for samples of size 20? In sampling from a norma1 population compare the average length of the two confidence intervals for the mean p. when (a) a is known and (b) a is unknown. Show that the length and the variance of the length of the t confidence interval for p. when sampling from a normal population approach 0 with increasing sample size. In sampling from a normal population with both p. and a unknown, how large a sample must be drawn-to make the probability .95 that a 90 percent confidence interval for p. will have length less than a15? Show that the length of the confidence interval for a of a norma1 population approaches 0 with increasing sample size. To test two promising new lines of hybrid com under norma1 farming conditions, a seed company selected eight farms at random in Iowa and planted both lines in experimental plots on each farm. The yields (converted to bushels per acre) for the eight locations were Line A: 86 87 56 93 84 93 75 79 Line B: 80 79 58 91 77 82 74 66 Assuming that the two yieldS are jointly normally distributed, estimate the difference between the mean yields by a 95 percent confidence interval.

400

PARAMETRIC INTERVAL ESTIMATION

VIU

19 X., ... , X,. is a random sample from (l/6)X(1-6)/6/ro . J](x), where 0 O. Find a l00y percent confidence interval for O. Find its expected length. Find the limiting expected length of your confidence interval. Find n such that P[length (50] p for fixed (5 and p. (You may use the central-limit theorem.) 20 Develop a method for estimating the parameter of the Poisson distribution by a confidence interval. 21 Find a good l00y percent confidence interval for 0 when sampling fromf(x; (J) = 1(6-1:.6+1:)(x). 22 Find a gOod tOOy percent confidence interval for 0 when sampling fromf(x; 0) = (2X/02)/(O.6)(X), where 0 > O. 23 One head and two tails resulted when a coin was tossed three times. Find a 90 percent confidence interval for the probability of a head. 24 Suppose that 175 heads and 225 tails resulted from 400 tosses of a coin. Find a 90 percent confidence interval for the probability of a head. Find a 99 percent confidence interval. Does this appear to be a true coin? 25 Let Xl, ... , X" be a random sample fromf(x; 6) = f(x; p., 0") = cPp,.02(X). Define T(J) by f:6) cPp,.a 2 (x) dx = a(a is fixed). Recall what the UMVUE of T(O) is. Find a l00y percent confidence interva1 for T(O). (If you cannot find an exact lOOy percent confidence interval, find an approximate one). 26 Let Xl, ... , Xn be a random sample from f(x; (J) = cP6.I(X). Assume that the prior distribution of@ is N(lLo,~),lLoandO"~known. Find a lOOypercentBayesian interval estimator of 0, and compare it with the corresponding confidence interval. 27 Let Xl, "', XII be a random sample from f(x 16) = OX'-I 1(0. H(X), where 0> O. Assume that the prior distribution of @ is given by
De(O)

= r(r) ){Or-I e -A6[(o. CC)(J),

where r and A are known. Find a 95 percent Bayesian interval estimator of O. *28 Let X denote the life in hours ofa radioactive particle. Suppose Xhas a density
f(x; 0) = Oe-'X[{o. oolx).

A random sample of n particles is put under observation, but the experiment is to stop when the kth particle has expired; i.e., it is intended not to wait until all the particles have ceased activity but only until k of them (k fixed in advance) have done so. The data consist of the k measurements Yt , " Yk and n - k measurements known only to exceed Y", where Y is the lifetime of the ith particle to expire. Find the maximum-likelihood estimator of the mean lifetime I/O. Also find a confidence-interval estimator of I/O.
j

IX
TESTS OF HYPOTHESES

1 INTRODUCTION AND SUMMARY

There are two major areas of statistical inference: the estimation of parameters and the testing of hypotheses. We shall study the second of these two areas in this chapter. Our aim will be to develop general methods for testing hypotheses and to apply those methods to some common problems. The methods developed will be of further use in later chapters. In experimental research, the object is sometimes merely to estimate parameters. Thus one may wish to estimate the yield of a new hybrid line of corn. But more often the ultimate purpose will involve some use of the estimate. One may wish, for example, to compare the yield of the new line with that of a standard line and perhaps recommend that the new line replace the standard line if it appears superior. This is a common situation in research. One may wish to determine whether a new method of sealing light bulbs will increase the life of the bulbs, whether a new germicide is more effective in treating a certain infection than a standard germicide, whether one method of preserving foods is better than another insofar as retention of vitamins is concerned, and so on.

402

TESTS OF HYPOTHESES

IX

Using the light-bulb example as an illustration, let us suppose that the average life of bulbs made under a standard manufacturing procedure is 1400 hours. It is desired to test a new procedure for manufacturing the bulbs. The statistical model here is this: We are dealing with two populations of light bul bs: those made by the standard process and those made by the proposed process. We know (from numerous past investigations) that the mean of the first population is about 1400. The question is whether the mean of the second population is greater than or less than 1400. Traditionally, to answer this type of question, we set up the hypothesis that one mean is greater than the other mean. Then, on the basis of a sample from the population of the proposed process we shall either accept or reject the hypothesis. F or our example, we formulate the hypothesis that the proposed process is no better than the standard process. Generally we hope that the hypothesis will be rejected. To test the hypothesis, a number of bulbs are made by the new process and their lives measured. Suppose that the mean of this sample of observations is 1550 hours. The indication is that the new process is better, but is 125 suppose that the estimate of the standard deviation of the mean rtl (n being the sample size). Then a 95 percent confidence interval for the mean of the second population (assuming normality) is roughly 1300 to 1800 hours. The sample mean 1550 could very easily have come from a popUlation with mean 1400. We have no strong grounds for rejecting the hypothesis. If, on the other hand,

In

rtl In were 25, then we could very confidently reject the hypothesis and pronounce
the proposed manufacturing process to be superior. The testing of hypotheses is seen to be closely related to the problem of estimation. It will be instructive, however, to develop the theory of testing independently of the theory of estimation, at least in the beginning. In order to conveniently talk a bout testing of hypotheses, we need to introduce some language and notation and give some definitions. As was the case when we studied estimation, we will assume that we can obtain a random sample Xl' ... , Xn from some density f( ; lJ). A statistical hypothesis will be a hypothesis about the distribution of the population.

Definition 1 Statistical hypothesis A statistical hypothesis is an assertion or conjecture about the distribution of one or more random variables. If the statistical hypothesis completely specifies the distribution, then it is called simple; otherwise, it is called composite. IIII

Notation To denote a statistical hypothesis, we will use a script capital

J'f followed by a colon that in turn is followed by the assertion that

specifies the hypothesis.

IIII

INTRODUCTION AND SUMMARY

403

EXAMPLE 1 Let Xl' ... , Xn be a random sample from I(x; fJ) = CPo, 2S(X), The statistical hypothesis that the mean of this normal population is less than or equal to 17 is denoted as follows: Yf: fJ < 17. Such a hypothesis is composite; it does not completely specify the distribution. On the other hand, the hypothesis Yf: fJ = 17 is simple since it completely specifies the distribution. IIII

Definition 2 Test of a statistical hypothesis . A test of a statistical hypothesis Yf is a rule or procedure for deciding whether to reject Yf. IIII Notation
Let us use a capital upsilon 1 to denote a test.

IIII

EXAMPLE 2 Let Xl' ... , Xn be a random sample from I(x; fJ) = CPo, 2S(X), Consider Yf: fJ ,:5;; 17. One possible test 1 is as follows: Reject Yf if and only if X > 17

+ 51Jn.

/III

A test can be either randomized or nonrandomized. The test 1 given in Example 2 above is an example of a nonrandomized test. Another possible test, say 1', of Yf in Example 2 is the following: Toss a coin, and reject Yf if and only if a head appears. Such is an example of a randomized test. Although we will make little use of randomized tests in this book, we do include their definition. Definitions of both nonrandomized and randomized tests follow.

Notation As in previous chapters, we let X denote the sample space of observations, or the potential data set; that is, X = {(Xl,., X n ): (Xl"'" Xn) is a possible value of (Xl' ... , X n)} IIII Definition 3 Nonrandomized test and critical region Let a test 1 of a statistical hypothesis Yf be defined as follows: Reject Yf if and only if (Xl, ... , xn) E Cy , where Cy is a subset of X; then 1 is called a nonrandomized test, and Cy is called the critical region of the test 1. IIII

EXAMPLE 3 Let Xl, "" Xn be a random sample from I(x; fJ) = CPo, 25(X), X is euclidean n space. Consider Yf: fJ < 17 and the test 1: Reject Yf if and on)y if x > 17 + 51 J~. Then 1 is nonrandomized, and Cy = {(Xl' ... , X n ): x > 17 + 5IJ~ IIII

}.

404

TESTS OF HYPOTHESES

IX

Remark A nonrandomized test Y of Jf is a decomposition of l: into C r

and Cr such that if(x1, ... , xn) E C r , Jf is rejected. So a nonrandomized fill test is specified by its corresponding critical region.

Definition 4 Randomized test A test Y of a hypothesis Jf is defined to be a randomized test if Y is defined by the function t/Jy(X1' "', xn) = P[Jf is rejected I (Xl' " ' , xn) is observed]. The function t/Jy(', "', .) is //11 called the critical/unction of the test Y.
The actual performing of a nonrandomized test Y of Jf is straightforward; one observes a random sample, say Xl' , Xn ,checks to see whether the observed sample falls in the critical region, and rejects Jf when it does. On the other hand, to perform a randomized test Y of Jf, one first observes the random sample, say Xl' , X n , then evaluates t/Jy(x b . " x n), and finally observes the result of some auxiliary Bernoulli trial that has t/Jy(x 1, .. " xn) as its probability of success, and if the Bernoulli trial resu.lts in a success, then Jf is rejected. Since the performance of the auxiliary Bernoulli trial is extraneous to the actual testing problem, that is, it does not depend on the data Xl' " Xn of the experiment, one might reasonably wonder why its result should be the deciding factor in accepting or rejecting the hypothesis. It is for this reason that r rriized tests are not often employed in practice; when they are, usuall e sample space l: is decomposed into three sets, one where the hypothesis is accepted, another where the hypothesis is rejected, and the third where" randomization" takes place. This third region is often the boundary between the acceptance and rejection region and/or a region where it is not easy to decide whether to accept or reject. The following example may help in understanding randomized tests. EXAMPLE 4 Let Xb "', X 10 be a random sample of size 10 fromf(x; lJ) = lJx(I - lJ)l- x for X = 0 or 1. Suppose we want to test the hypothesis
Jf: lJ < Jf if

t.

A possible test Y is the following: Reject Jf if I Xi > 5, accept

1

10

10

Xi < 5, and decide between rejecting and accepting by tossing a

10
1

fair coin if L Xi

5.

(The tossing of the coin is the auxiliary Bernoulli

trial.) Such a test Y partitions l: into three regions, say A, B, and C, where

A
B

{(X" "', X,o):

~ X; < 5},
~ XI = 5},

= {(X" ... , X,o):

INTRODUCTION AND SUMMARY

405

and

c {(x" ... , XlO): ~ Xi> 5}.

The critical function of test T is given by 1
t/lr(Xl' . , XI0)

if (Xb if (Xb

... , XI0) E , X 10) E

= 1/2

IIII

Remark We saw that a nonrandomized test was specified by its critical

region. Likewise, a randomized test is specified by its critical function. In fact any function t{t(., .,., .) with domain l! and counterdomain the interval [0, I] is a possible critical function and defines a randomized
~

The following remark shows that a nonrandomized test is a particular case of a randomized test.

Remark If test T has a critical function defined by

1/IY{X" ... , xJ

= {~

if (Xl' , x n)' . otherwlse

r) -- 1

(
r

Xl"'"

Xn ,

then T is a nonrandomized test with a critical region Cr.

IIII

As we mentioned earlier, we will not make extensive use of randomized tests. Theorem I below requires their use; other than that, their only use will be in obtaining tests of exact size (see Definition 7), and then only for sampling from discrete distributions. In many hypotheses-testing problems two hypotheses are discussed: The first, the hypothesis being tested, is called the null hypothesis, denoted by :7t 0 , and the second is called the alternative hypothesis, denoted by :7t l' The thinking is that if the null hypothesis is false, then the alternative hypothesis is true, and vice versa. We often say that :7t 0 is tested against, or versus, :7tI' If the null hypothesis :7t 0 is not rejected, we say that :7t0 is accepted, With this kind of thinking, two types of errors can be made. Definition 5 Types of error and size of error Rejection of :7t0 when it is true is called a Type I error, and acceptance of :7t0 when it is false is called a Type II error. The size of a Type I error is defined to be the probability that a Type I error is made, and similarly the size of a Type II error is the IIII probability that a Type II error is made.

406

TESTS OF HYPOTHESES

IX

If the distribution from which the sample was obtained is parameterized by 0, where 0 E f), then associated with any test is a power function, defined as in Definition 6.

Definition 6 Power function Let Y be a test of the null hypothesis .7t 0 + The power function of the test Y, denoted by 1ty(O), is defined to be the probability that .7t 0 is rejected when the distribution from which the sample was obtained was parameterized by O. IIII
The power function will play the same role in hypothesis testing that meansquared error played in estimation. It will usually be our standard in assessing the goodness of a test or in comparing two competing tests. An ideal power function, of course, is a function that is 0 for those 0 corresponding to the null hypothesis and is unity for those 0 corresponding to the alternative hypothesis. The idea is that you do not want to reject .7t 0 if .7t 0 is true and you do want to reject .7t 0 when .7t 0 is false.

Remark 1ty(O) = P8[reject .7t 0]' where 0 is the true value of the parameter. If Y is a nonrandomized test, then 1ty(O) = P8 [(X1 , , Xn) E Cy], where Cy is the critical region associated with test Y. If Y is a randomized test with critical function t/ly (. , ... , +), then
1ty(O) = P 8[reject.7t 0]
=

f f

P[reject .7t 0 I Xl'

X n]!Xl .... Xn(X 1 , , Xn ;

0)

1=

Ii

dXi

cf8 [t/ly(X I, + +., X n)] + ,

The argument is similar for discrete

ran~/variables+
=

IIII

EXAMPLE 5 Let Xb +", Xn be a random sample fromf(x; 0) Consider .7t0:

cP8.2S(X),

0<

17 and the test Y: Reject if and only if X > 17

+ 51 J~.

1ty(0) = P 8 [ X > 17

+ J~ =

5 ]

I - <I>

(17 +

51J~ 51J n

0) .

For n = 25, 1ty(O) is sketched in Fig. 1.

INTRODUCTION AND SUMMARY

407

1.0

.5

FIGURE 1

~----~----~----~~~~-----8

The power function is useful in telling how good a particular test is. In this example, if 0 is greater than about 20, the test Y is almost certain to reject :J'f 0' as it should. And if 0 is less than about 16, the test Y is almost certain not to reject :J'f 0, as it should. On the other hand, if 17 < 0 < 18 (so :J'f 0 is false), the test Y has less than half a chance of rejecting :J'f 0 II!!

Definition 7 Size of test Let Y be a test of the hypothesis :J'f0: 0 E eo, where eo c e; that is, eo is a subset of the parameter space S. The size of the test Y of :J'f0 is defined to be sup [nr(O)]. The size of the test for
8e~o

a nonrandomized test is also referred to as the size of the critical region.

IIII
Remark Many writers use the terms "significance level" and "size of test" interchangeably. We, however, will avoid use of the term" significance level/' intending to reserve its use for tests of significance, a type of statistical inference that is closely related to hypothesis testing. Tests of significance will not be considered in this book; the interested reader is referred to Ref. [37]. IIII
EXAMPLE 6 Let Xl' "" Xn be a random sample from f(x; 0) = 4>0. 2S(x). Consider the :J'f 0 = 0 < 17 and the test Y: Reject :J'f0 if X > 17 + 51Jn.
~o = {O: 0 < 17} and the size of the test Y is sup [nr(O)]
_
Oe~o

= sup
Osl7

[I - (J)(17 +51 5~n -6)] = 1 - (J)(l) ~ .159. n

IIII

In our study of point estimation, we found that for certain considerations we could restrict attention to estimators that were functions of sufficient statistics only. The same is true for testing hypotheses when the power function is used as a basis of comparing tests, as the following theorem shows.

408

TESTS OF HYPOTHESES

IX

Theorem 1 If Xl' ... , X 1I is a random sample from/ex; e), where e E 9, and Sl = 61(X1, ... , X1I ) , , S,. = 6,.(X1, ... , X1I ) is a set of sufficient statistics, then for any test i with critical function 0/1' there exists a test, say if, and corresponding critical function, say 0/1', depending only on the set of sufficient statistics which satisfies 1t1 (e) = 1t1'(e) for all e E 8.
Define o/l'(Sb "" S,.) = 8[0/1(X1, ... , X1I ) lSI = Sl, ... , S,. = S,.]; then 0/1' is a critical function. Furthermore, 1t1'(e) = 4 9 [o/y,(Sl' ... , S,.)] = 8 9 [8[0/1(X1, ... , X1I ) ISb . ", S,.]] = 8 9 [I/II(X1, ... , X 1I) ] = 1ty(e). IIII
PROOF

The theorem shows that given any test, another test which depends only on a set of sufficient statistics can be found, and this new test has a power function identical to the power function of the original test. So, in our search for good tests we need only look among tests that depend on sufficient statistics. We have introduced some of the language of testing in the above. The problem of testing is like estimation in the sense that it is twofold: First, a method of finding a test is needed, and, second, some criteria for comparing competing tests are desirable. Although we will be interested in both aspects of the problem, we will not discuss them in that order. First we will consider, in Sec. 2, the problem of testing a simple null hypothesis against a simple alternative. Two approaches will be assumed. The first will use the power function as a basis for setting goodness criteria for tests, and the second will use a loss function. The Neyman-Pearson lemma is stated and proved. It will turn out that all those tests, which are best in some sense, will be of the form of a simple likelihood-ratio, which is defined. Tests of composite hypotheses will be discussed in Sec. 3. The section will commence, in Subsec. 3.1, with a discussion of the generalized likelihoodhis principle plays a ratio principle and the generalized likelihood-ratio test. ~. central role in testing, just as maximum likelihood played central role in estimation. It is a technique for arriving at a test that in gen ral will be a good test, just as maximum likelihood led to an estimator that in general was quite a good estimator. For a book of the level of this book, it is probably the most important concept in testing. The notion of uniformly most powerful tests will be introduced in Subsec. 3.2, and several methods that are sometimes useful in finding such tests will be presented. Unbiasedness and invariance in estimation are two methods of restricting the class of estimators wi th the hope of finding a best estimator within the restricted class. These two concepts play essentially the same role in testing; they are methods of restricting the totality of

SIMPLE HYPOTHESIS VERSUS SIMPLE ALTERNATIVE

409

possible tests with the hope offinding a best test within the restricted class. We will discuss only unbiasedness, and it only briefly in Subsec. 3.3. Subsection 3.4 will summarize several methods of finding tests of composite hypotheses. Section 4 will be devoted to consideration of various hypotheses and tests that arise in sampling from a normal distribution. Section 5 will consider tests that fall within a category of tests generally labeled chi-square tests. Included will be the asymptotic distribution of the generalized likelihood-ratio, goodness-of-fit tests, tests of the equality of two or more distributions, and tests of independence in contingency tables. Section 6 will give the promised discussion of the connection between tests of hypotheses and interval estimation. The chapter will end with an introduction to sequential tests of hypotheses in Sec. 7. The reader will note that our discussion of tests of hypotheses is not as thorough as that of estimation. Both testing and estimation will be used in later chapters, especially in Chap. X. Also, a number of the nonparametric techniques that will be presented in Chap. XI will be tests of hypotheses. We stated at the beginning of this section that testing of hypotheses is one major area of statistical inference. A type of statistical inference that is closely related (in fact so closely related that many writers do not make a distinction) to hypothesis testing is that of significance testing. The concept of significance testing has important use in applied problems; however, we will not consider it in this book. The interested reader is referred to Ref. [37].

SIMPLE HYPOTHESIS VERSUS SIMPLE ALTERN A TIVE Introduction

2.1

In this section we consider testing a simple null hypothesis against a simple alternative hypothesis. This case is actually not very useful in applied statistics, but it will serve the purpose of introducing us to the theory of testing hypotheses. We assume that we have a sample that came from one of two completely specified distributions. Our object is to determine which one. More precisely, assume that a random sample Xl"'" Xn came from the density fo(x) or fl (x) and we want to test :Yf 0 : Xi distributed asJo('), abbreviated Xi '" fo{'), versus :Yf1: Xi '" It ( .). If we had only one observation Xl and foe .) and It (.) were as in Fig. 2, one might quite rationally decide that the observation came from fo(') if fo(x 1) > Ji(Xl) and, conversely, decide that the observation came from It (.) if fi (Xl) > fo(xl)' This simple intuitive method of obtaining a test can be expanded into a family of tests that, as we shall see, will contain some good tests.

410 TESTS OF HYPOTHESES

IX

FIGURE 2

Definition 8 Simple likelihood-ratio test Let Xl' ... , Xn be a random sample from either fo{') or It (.). A test Y of :Yf 0: Xi""" fo{ .) versus :Yf1 : Xi""" fi(') is defined to be a simple likelihoodMratio test ifY is defined by
Reject :Yfo if A < k, Accept :Yf 0 if A > k, Either accept :Yf 0, reject :Yf 0, or randomize where

if A = k,

(1)

and k is a nonnegative constant. [L j = Lixl' ... , xn) is the likelihood 1111 function for sampling from the density Ii' ).] For each different k we have a different test. For a fixed k the test says to reject :Yf 0 if the ratio of likelihoods is small; that is, reject :Yf0 if it is more likely (Ll is large compared to Lo) that the sample came from fi ( .) than from lo{ . ). Such a test certainly has intuitive appeal. In fact, one might suspect that an optimum test will have to be the form of a simple likeliho?dMr.ati~.~ Optimality of a test of a simple hypothesis versus asimpLe-atrernative can be approached in two ways. One way, using the power of the test to set goodness criteria, is discussed in SUbsec. 2.2, and the other way, using a loss function and a decisionMtheoretical approach, is considered in Subsec. 2.3.

2.2 Most Powerful Test

Let Xl' .. , Xn be a random sample from the density fo{') or the density 11 ('). Let us write lo(x) = f(x; ( 0 ) and It (x) = f(x; ( 1 ); then Xl' ... , Xn is a random sample from one or the other member of the parametric family {/(x; 0): = 00 orO = Ol}' = {Oo, 0l} is a parameter space with only two points in it. 00 and

SIMPLE HYPOTHESIS VERSUS SIMPLE ALTERNATIVE

411

(}l are known. We want to test :tt 0 : (} = (}o versus :ttl : (} = (}l' Corresponding to any test 1 of :tt 0 versus :ttl is its power function 1'ly(8). A good test is a test for which 1ty(}o) = P[reject :tt 0 l:tt 0 is true] is small (ideally 0) and 1'ly(81) = P[reject -*'ol:tfo is false] is large (ideally unity). One might reasonably use the two values 1ty(Jo) and 1'lY(}I) to set up criteria for defining a best test. 1'ly(}o) = size of Type I error, and I - 1'lY(}I) = P[accept:tt 0 I:tt 0 is false] = size of Type II error; so our goodness criterion might concern making the two error sizes small. F or example, one might define as best that test which has the smallest sum of the error sizes. Another method of defining a best test, made precise in the following definition, is to fix the size of the Type I error and to minimize the size of the Type II error. Definition 9 Most powerful test A test 1* of :tt 0: (} = (}o versus :ttl: (} = (}l is defined to be a most powerful test of size (1 (0 < (1 < 1) if and only if:
(i)

(ii)

1'ly-(}o) = (1. 1'lY-(}I) > 1'lY(}I) for any other test 1 for which 1'ly(}o) <

(1.

(2) (3)

1/11
A test 1* is most powerful of size (1 if it has size (1 and if among all other tests of size (1 or less it has the largest power. Or a test 1* is most powerful of size (1 if it has the size of its Type I error equal to (1 and has smallest size Type II error among all other tests with size of Type I error (1 or less. The justification for fixing the size of the Type I error to be (1 (usually small and often taken as ,.05 or .01) seems to arise from those testing situations where the two hypotheses are formulated in such a way that one type of error is more serious than the other. The hypotheses are stated so that the Type I error is the more serious, and hence one wants to be certain that it is small. The following theorem is useful in finding a most powerful test of size (1. The statement of the theorem as given here, as well as the proof, considers only nonrandomized tests. We might note that the statement and proof of the theorem can be altered to include all randomized tests. Theorem 2 Neyman-Pearson lemma Let Xl, ... , Xn be a random sample from f(x; (}), where (} is one of the two known values (}o or (}l' and let 0 < (1 < 1 be fixed. Let k* be a positive constant and C* be a subset of ~ which satisfy:
(i)

P8o[(XI, ... , Xn)

C*] =

(1.

(4)

412

TESTS OF HYPOTHESES

IX

..) (11

/ .. -

~ _ L(Oo; Xl' ... ,

L(

1;

Xb

XJ _ LO ... , XJ Ll
E

k*

(5)

and )" > k* if (Xb ... , xn)

C*.

Then, the test y* corresponding to the critical region C* is a most powerful = 1 , [Recall that L) test of size ct of Jf 0: 0 versus Jf 1 :
n

L(OJ; Xl' ... , Xn)

i= 1

n f(Xi; OJ) for j = 0 or 1 and C* is the complement of

0=

C*; that is, C* = l - C*.]

Suppose that k* and C* satisfying conditions (i) and (ii) exist. If there is no other test of size ct or less, then y* is automatically most powerful. Let Y be another test of size ct or less, and let C be its corresponding critical region. We have P8o[(X1, ... , Xn) E C] < ct. We must show that 1ly.(Ol) > 1ly(Ol) to complete the proof. {For any subset
PROOF

R of X, let us abbreviate

J.~. J (,V, I(x,; 8J) dx,]

as

JR L J for j

0, 1.

Our notation indicates that fo( .) and It ( . ) are probability density functions. The same proof holds for discrete density functions.} Showing that 1ly.(01 ) > 1ly(Ol) is equivalent to showing that Ic. Ll > Ic L 1 See Fig. 3. Now Ic. L l - Ic L l = Ic.c Ll - Icc. L l > (llk*) Ic*cLo- (llk*) Icc*L o sinceLl > Lo/k* on C* (hence also on C*C) and Ll < Lo/k*, or -Ll -Lo/k*, on C* (hence also on CC*). But (l/k*) (Jc*c Lo - Icc. Lo)
- Ic.cLo - Icc.Lo) = (llk*)(Ic*L o - IcLo) = (l/k*)(ct - size of test Y) > 0; so Ic. Ll - Ic Ll > 0, as was to be shown.

= (llk*){Jc.cLo + Ic*cLo

IIII
We comment that k* and C* satisfying conditions (i) and (ii) do not always exist and then the theorem, as stated, would not give a most powerful size-ct test. However, whenever fo( .) and It ( .) are probability density functions, a k* and C* will exist. Although the theorem does not explicitly say how to find k* and C*, implicitly it does since the form of the test, that is, the critical region, is given by Eq. (5). In practice, even though k* and C* do exist, often it is not necessary to find them. Instead the inequality A S; k* for (Xh ... , xn) E C* is manipulated into an equivalent inequality that is easier to work with, and the actual test is then expressed in terms of the new inequality. The following examples should help clarify the above.

SIMPLE HYPOTHESIS VERSUS SIMPLE ALTERNATIVE

413

FIGURE 3

EXAMPLE 7 Let Xl"'" Xli bea random sample from/ex; (J) = (Je- /(0. oo)(x), where (J = 80 or 8 = 81 , (Jo and (JI are known fixed numbers, and for concreteness we assume that (JI > (Jo' We want to test .Yf0: (J = (Jo versus .Yfl : (J = (JI' Now Lo = 03 exp (-(Jo LXi), L1 = 0i exp (-(JI I Xi), and according to the NeymanftPearson lemma the most powerful test will have the form: Reject .Yf0 if )" < k* or if (JO/(JI)'J exp [ - (Jo - (JI) I < k*, which is equivalent to .

8X

xa

LXi < 0, ~ 00 log. [ (::) nk*]

k' (say),

where k' is just a constant. The inequality A =:;; k* has been simplified and expressed as the equivalent inequality L Xi k'. Condition (i) is ex = PfJo[reject .Yfol = PfJo[ I Xi =:;; k']. We know that Xi has a gamma distribution with parameters n and 0; hence

p fJo [ f..J " Xl -< k'] =

I fr(n)
k'
0

(In
0

X'- l e- x8o dx = ex ,

an equation in k', from which k' can be determined; and the most powerful test of size ex of .Yfo: (J = (Jo versus .Yfl : (J = (JI' (JI > (Jo is this: Reject .Yfo if LXi k', where k' is the exth quantile point of the gamma distribution with parameters nand (Jo 11// EXAMPLE 8 Let Xl' ... , Xn be a random sample from /(x; (J) = (Jx(1 - (J)l-X I{O.I}(X), where (J = (Jo or (J = (J1' We want to test .Yf0: (J = 00 versus .Yf1 : 0 = (Jl' where, say, (Jo < 01 , Lo = (J~Xi(1 - (Jo)n-r.xi, Ll = (Jr Xt (1 Odn-r.xt, and so A < k* if and only if
(J~Xi(1 -

(JO)'J-r.Xi/(J

Xi(l - (Jl)n-r. Xi

< k*,

('-

if and only if

414

TESTS OF HYPOTHESES

IX

or if and only if I Xi > k', where k' is a constant. {Note that loge [0 0 0 - ( 1 )/0 - ( 0 )01 ] < O.} So a most powerful test would be of the form: Reject :tf 0 if I Xi is large. For definiteness, let us take 0 0 = !, 01 = i, and n = 10. We must find k' so that
(X=P80=1/4[reject:tfO]=P80=1/4[IXi:C:k']

(10) (1) i (3) 10 - i I . . i =k' 1 4 4

10

If (X = .0197, then k' = 6, and if (X = .0781, then k' = 5. For (X = .05, there is no critical region C* and constant k* of the form given in the NeymanPearson lemma. In this example our random variables are discrete, and for discrete random variables it is not possible to find a k* and C* satisfying conditions (i) and (ii) for an arbitrary fixed 0 < (X < 1. In practice, one is usually content to change the size of the test to some (X for which a test given by the Neyman-Pearson lemma can be found. We might note, however, that a most powerful test of size (X does exist. The test would be a randomized test. For the example at hand, if we take (X = .05, the randomized test with critical function

1
.0584

o
is the most powerful test of size
(X

= .05.

IIII

In closing this subsection, we note that a most powerful test of size (x, given by the Neyman-Pearson lemma, is necessarily a simple likelihood-ratio test.

2.3

Loss Function

As in the last subsection, we assume that we have a random sample Xl, ... , Xn from one or the other of the two completely known densities fo( .) = f( . ; ( 0 ) and it ( .) = f( ; ( 1 ), On the basis of the observed sample we have to decide from which of the two densities the sample came; that is, we test :tf 0: 0 = 00 versus :tf1: 0 = 01, We can make one of two decisions, say do or d 1, where d j is the decision that fi') is the density from which the sample came, j = 0, 1. We assume that a loss function is available.

Definition 10 Loss function In testing :tf 0: 0 = 0 0 versus :tf1 : 0 = Ob define t(d i ; OJ) = loss incurred when decision d i is made and OJ is the true

SIMPLE HYPOTHESIS VERSUS SIMPLE ALTERNATIVE

415

parameter value for i = 0, 1 andj = 0, 1, where dE is the decision of deciding that hypothesis :Yf i is correct. We will adopt the convention that ted;; (Ji) = 0 for i = 0, 1 and t(di ; OJ) > 0 for i #=j. IIII The values of the loss function are the amounts that are lost if decision dE is made when (J. was correct. With our convention, nothing is lost if the right J decision is made, and a positive amount is lost if a wrong decision is made. If we think in terms of a (nonrandomized) test Y having a critical region Cr , then decision d1 is made if the observed sample (Xl' ... , XII) belongs to C r, and decision do is made if (Xl' .. , xn) belongs to Cr A test can be thought of as a decisionfunction since for a given observed sample the test tells us which decision to make. We do not consider the problem of selecting an appropriate loss function, and hence we will always assume that an appropriate loss function has been prescribed. Note that if decision d1 is made when 00 is correct, then a Type I error is made, and if decision do is made when 01 is correct, then a Type IT error is made. In comparing two tests we naturally prefer that test which has smaller loss, and among all tests we would prefer that test which has smallest loss. However, seldom will there exist one test that has smallest loss for both possible decisions and for both 00 and 01 , This motivates the defining of average loss, and by continuing to borrow language from decision theory we define the risk function of a test.
Definition 11 Risk function For a random sample X h " ' , Xn from f(' ; 00 ) or f( ; 01), let Y be a test of :Yf 0: 0 = 00 versus :Yf 1: (J = 01 having a critical region Cr. For a given loss function t(; .), the risk function of 1, denoted by !!lr(O), is defined to be the expected loss; that is

Remark aly(O) = t(d1 ; 0)Pe[(X1 ,

XJ E Cr ] + t(do ; 0)Pe[(X1 , = t(d1 ; O)nr(O) + t(do; 0)[1 - nr(O)];

,

Xn)

(;r]

(6)

that is, the risk function is a linear function of the power function; the coefficients in the linear function are determined by the values of the loss function. Since 0 assumes only two values, !!lr(O) can take on only two values, which are and
al y{Ol)

= t(do ; 01)[1

- n 1 (01)]'

(7)

IIII

416

TESTS OF HYPOTHESES

IX

Our object is to select that test which has smallest risk, but, unfortunately, such a test will seldom exist. The difficulty is that the risk takes on two values, and a test that minimizes both of these values simultaneously over all possible tests does not exist except in rare situations. (See Prob. 3.) Not being able to find a test with smallest risk, we resort to another less desirable criterion, that of minimizing the largest value of the risk function.

Definition 12 Minimax test A test 1m of ff 0: fJ = (}o versus ff 1: (} = fJ1 is defined to be minimax if and only if
max [&frm(fJO ), &f rm (OI)] < max [&fr(fJo), &fr (}I)] for any other test 1.

IIII

The following theorem is sometimes useful in finding a minimax test. (As with Theorem 2, we state the theorem and proof in terms of nonrandomized tests.)

Theorem 3 For a random sample Xl' ... , X" from f(; ( 0 ) orf( ; ( 1 ),
consider testing ff 0: fJ = 00 versus ff 1 : = 1 , If a test Ymhas a critical region given by Cm = {(Xl' ... , xJ: A <km }, where k m is a positive constant ) = &f rm(}I), then 1m is minimax. Recall that such that &frm(fJ O

A = LolLI =

" f(XI; (}o)]/[ I1 " f(Xi; ( 1)], [I1

1= I i
I

PROOF We will assume that f(; fJ o) and f(; ( 1 ) are probability density functions. The proof for discrete density functions is similar. Let 1 be any other test with a critical region Cr which satisfies &fr(Oo) < &frm(Oo). [Note that jf &fr(}o) > &frm(}o), then Y would not even be a candidate for minimax.] We have &fr(fJ o) < &frm(}o), t(d1 ; 0o)1tr(}o) < t(dl ; Oo)1tr m(fJ O) , or 1tr(}o) 1trm(fJO); that is, 1 has size less than or equal to that of 1m' But, by the Neyman-Pearson lemma, Ym is the most powerful test of size 1tl'm(OO); hence 1tr(OI) < 1trm(OI), 1 1tr(OI) > 1 - 1trm(}I), t(do ; 0d[l - 1tr(fJd] > t(do ; fJ1)[l - 1trm(fJI)], or &fr(OI) > &frm(OI); so, we have max [,!{lrm(}o), &frm (}I)] = &f rm(fJ1 ) < ,!{lY{(}I) < max [&fr(}o), &fr(fJd]; that is, 1m is minimax. IIII EXAMPLE9 Let Xl, ... , X" be a random sample fromf(x; fJ) = fJe- 8xI(o, oo)(x), For 01 > 0 , test ffo: 0= 00 versus ff l : = 1 , In Example 7, we found the most powerful size-tX test. We seek now to find the minimax test for a loss function given by t(d1 ; fJ o) = a and t(do ; fJ1) = h. According to Theorem 3, the minimax test 1m is given by Cm = {(Xl' ... , x,,): A k m }

SIMPLE HYPOTHESIS VERSUS SIMPLE ALTERNATIVE

417

Before leaving minimax tests we make two comments: First, if 10(') and 11 ( .) are discrete density functions, then there may not exist a k m such that r71 ym(()o) = r71 Ym (()l) unless randomized tests are allowed; and, second, a minimax test as given in Theorem 3 was a simple likelihood-ratio test. In the above we assumed that Xl, ... , Xn was a random sample from 1(' ; (J), where () = ()o or ()l' and for each (), 1(' ; ()) is completely known. We also assumed that we had an appropriate loss function. Now, we further assume that ()o and ()l are the possible values of a random variable e and that we know the distribution of e, which is called the prior distribution, just as in our considerations of Bayes estimation. e is discrete, taking on only two values ()o and ()l; so the prior distribution of e is completely given by, say, g, where 9 = p[e = ()l] = 1 - p[e = ()o]. We mentioned above that, in general, a test with smallest risk function for both arguments does not exist. Now that we have a prior distribution for the two arguments of the risk function, we can define an average risk and seek that test with smallest average risk.

Definition 13 Bayes test A test Y9 of .Yf0: () = ()o versus .Yf1 : () = ()l is defined to be a Bayes test with respect to the prior distribution given by 9 = pre = ()l] if and only if
(1 - g)PAYg(()o)

+ gPAYg(()l) < (l

- g)PAy(()o)

+ gPAY(()l)

(8)

for any other test Y. To find a Bayes test we seek a critical region Cg that minimizes
(1 - g)PAy(()o)

fill

+ gPAY(()l) =

(1 - g)t(dl ; ()o)1Ly(()o)

+ gt(do ; ()l)[l

- 1Lr(()l)]

= (l
as a function of the region C.
(1 - g)t(dl ; (Jo)

- g)t(dl ; ()o)

$e Lo + gt(do ; ()l) $e Ll

Now

Ie Lo + gt(do ; ()l) $e Ll
=

gt(do ; (Jl) + $d(l - g)t(dl ; ()o)Lo - gt(do ; ()l)Ll ],

(9)

418

TESTS OF HYPOTHESES

IX

which is minimized if C is defined to be all (Xl' ... , xn) for which the last inte grand of Eq. (9) is negative; that is
Cg = {(Xb .. " x,,): (l - g)t(d1 ; Oo)Lo - gt(do ; (1)L 1 < O}.

(10)

We have proved the following theorem.

Theorem 4 The Bayes test Yg of .Yf'0: 0 = 00 versus .Yf'1: 0 = 01 with respect to a prior distribution. given by g = p[e = 01 ] has a critical region
defined by
(II)

IIII
We note that once again a good test, in this case a Bayes test, turns out to be a simple likelihood-ratio test. The exact form of the Bayes test is given by Eq. (11).

EXAMPLE 10 Let Xl, "', X" be a random sample from f(x; 0) = Oe -8x1(0, colx). Test.Yf' 0: 0 = 00 versus .Yf'1: (J = (Jl' The critical region of a Bayes test is given by
_ ( . 00 exp ( - 00 I C -l(Xl' .. " xn) (Ji exp (- 01 I
g

Xi) Xi)

gt(do ; (J1) } < (I - g)t(d ; (Jo)

1

1 (Xh ... , Xn): I

Xi

< (Jl

I
- (Jo

loge (1 - g)Oo t(d

g(Ji t(do ; ( 1 )
;( ) 1 0

for 01 >0 0 ,

IIII

3 COMPOSITE HYPOTHESES
In Sec. 2 above we considered testing a simple hypothesis against a simple alternative. We return now to the more general hypotheses-testing problem, that of testing composite hypotheses. We assume that we have a random sample fromf(x; 0), (J E 9, and we want to test .Yf'o: 0 E 9 0 versus.Yf'l: (J E 9f, where eo c e, e 1 c 8, and eo and e1 are disjoint. Usually 9 1 = 9 - 9 0 , We begin by djscussing a general method of constructing a test.

COMPOSITE HYPOTHESES

419

3.1

Generalized Likelihood-ratio Test

For a random sample Xl' .," X" from a densityf(x; 0), 0 E 9, we seek a test of Jff o : (J E versus Jff1 : 0 E 9 1 = 9 - 9 0 ,

eo

Definition 14 Generalized likelihood-ratio Let L(O; x h . , x,,) be the likelihood function for a sample Xl' ... , Xn having joint density fx." ... Xn (Xl' ... , x,,; 0) where 0 E 9. The generalized likelihood-ratio, denoted by 2 or An , is defined to be
(12)

1111
Note that 2 is a function of Xl' ... , x,,, namely 2(Xl' ... , x,,). When the observations are replaced by their corresponding random variables Xl, ... , X"' then we write A for 2; that is, A = 2(Xl' ... , X,,). A is a function of the random variables Xl, ... , X" and is itself a random variable. In fact, A is a statistic since it does not depend on unknown parameters. Several further notes follow: (i) Although we used the same symbol ), to denote the simple likelihood-ratio, the generalized likelihood-ratio does not reduce to the simple likelihood-ratio for = {Oo, Ol}' (ii) 2 given by Eq. (12) necessarily satisfies 0 < 2 < I; 2 >0 since we have a ratio of nonnegative quantities, and )" < I since the supremum taken in the denominator is over a larger set of parameter values than that in the numerator; hence the denominator cannot be smaller than the numerator. (iii) The parameter 0 can be vectorvalued. (iv) The denominator of A is the likelihood function evaluated at the maximum-likelihood estimator. (v) In our considerations of the generalized likelihood*ratio, often the sample X h . . . , X" will be a random sample from a density f(x; 0) where (J E 9. The values 2 of the statistic A are used to formulate a test of Jff 0 = 0 E 9 0 versus Jff1 : 0 E 9 - 9 0 by employing the generalized likelihood-ratio test principle, which states that Jff 0 is to be rejected if and only if 2 < 20 , where 20 is some fixed constant satisfying 0 < 20 < I. (The constant 20 is often specified by fixing the size of the test.) A is the test statistic. The generalized likelihoodratio test makes good intuitive sense since 2 will tend to be small when Jff 0 is not true, since then the denominator of 2 tends to be larger than the numerator. In general, a generalized likelihood-ratio test will be a good test; although there are examples where the generalized likelihood-ratio test makes a poor showing

420 TSSTS OF HYPOTHESES

IX

compared to other tests. One possible drawback of the test is that it iSlIOtn:ez times difficult to find sup L(O; Xb .. , x n); another is that it can be difficult to' find the distribution of A which is required to evaluate the power of the test.

EXAMPLE 11 Let Xl' ... , Xn be a random sample from f(x; 0) = Oe -8x/(0. oolx), where t:i = {o; 0 > O}. Test JIP 0: 0 < 00 versus JlPI : > 00

8eft

sUEL[(O;xH""Xn )] = sup
8>0

[On exp (-OIxi)]

= ( "n
~Xi

, )n e- n

and sup (L(O;

(I

Xl' . ,

Xn)]

= sup (on exp ( - 0 L Xl)]

0<8.s(lo

e.eo

(Inx)"e-.
~ exp (-0 0 LXi)

if --::;;;, 00 LXi if

r>Oo'
Xi

Hence
1

A=

if - - < 00 LXi -

O~ exp ( - 00 LXi)

(niI xi)ne-n

if

r >
Xi

(13)
0 ,

If 0 < Ao < 1, then a generalized likelihood-ratio test is given by the following: Reject JIP 0 if A < Ao , or reject JIP 0 if "'\'n > 00
~Xi

and

00 I n Xl)n exp ( -0 0 Ix; + n) < Ao,

(14)

or reject JIP 0 if Oox < 1 and (0 0 x)n e- n(8ox-l) < Ao. Write y = 00 x, and note that yne-n(.v-I) has a maximum for y = 1. Hence y < 1, and yne n(y-l) < Ao if and only if y < k, where k is a constant satisfying o < k < 1. See Fig. 4. We see that a generalized likelihood-ratio test reduces to the following: Reject JlPo if and only if Box < k, where 0 < k < 1; (15)

COMPOSITE HYPOTHESES

421

FIGURE 4

O~~--+---~l----------~==~-Y

that is, reject :Yf 0 if x is less than some fraction of 1/00 , If that generalized likelihood-ratio test having size a is desired, k is obtained as the solution to the equation "k 1 l U a = P9o[OoX < k] = P9o [Oo L Xi < nk] = 0 r(n) u"- e- duo

(Note that P9[OoX < k]

P9o [OoX < k] for 0 <0 0

,)

III/

We note that in the above example the first form of the test, as given in Eq. (14), is rather messy, yet after some manipulation the test reduces to a very simple form as given in Eq. (15). Such a pattern often appears in dealing with generalized likelihood-ratio tests-their first form is often foreboding, yet the tests often simplify into some nice form. We will observe this again in Sec. 4 below when we consider tests concerning sampling from the normal distribution. We might note, by considering the factorization criterion, that a generalized likelihood-ratio test must necessarily depend only on minimal sufficient statistics. In Sec. 5 below, a large-sample distribution of the generalized likelihoodratio is given. This will provide us with a method of obtaining tests with approximate size a. 3.2 Uniformly Most Powerful Tests

In Subsec. 3.1 above we exhibited a method of obtaining a family of tests of :Yf 0: () E E)o versus :lf l : () E E) - E)o. We now define one optimum property that such a test may possess. It is defined in terms of the power function 1'Cy(O) and the size of the test. Definition 15 Uniformly most powerful test A test y* of :Yf0: 0 E 8 0 versus :Ye l : 0 E e - E)o is defined to be a uniformly most powerful size-a test if and only if: (i)
(ii)

sup 1'Cy.(O)
gei!o

a.
E E) -

1'Cy.(ll) > 1'Cy(O) for all ()

8 0 and for any test Y with size

less than or equal to a.

/11/

422

TESTS OF HYPOTHESES

IX

A test y* is uniformly most powerful of size (X if it has size (X and if among all tests of size less than or equal to (X it has the largest power for all alternative values of 8. The adverb" uniformly" refers to " all" alternative 8 values. A uniformly most powerful test does not exist for all testing problems, but when one does exist, we can see that it is quite a nice test since among all tests of size (X or less it has the greatest chance of rejecting .Yf0 whenever it should.

EXAMPLE 12 Let Xl, ... , Xn be a random sample from I(x; 8) = 8e- 9x/(o.ooi x ), where 8 = {8: 8 >8 o}. Find a uniformly mOrtowerful test of .lito: 8 = 80 versus .Yf1 : 8 > 80 , For fixed 81 > 80 , we termined in Example 7 that the most powerful test of .Yf0: 8 = 80 versus 1: 8 = 81 was given by the following: Reject .lito if I Xi < k', where k' was given as a solution to the equation
(X

fo

k'

1
r(n)

8~ ~-1 e- 90x dx.

Such a test was given by the Neyman-Pearson lemma. Note that the test in no way depends on 81 except that 81 > 80 ; hence, we would get the same most powerful test for any 81 > 80 , and thus the test is actually uniformly /1/ / most powerful! The above example provides us with an example of a situation where a uniformly most powerful test can be obtained using the Neyman-Pearson lemma. That same technique can be used to find uniformly most powerful tests in more general situations, such as those given in Theorems 5 and 6 below, which are given without proof.

Theorem 5 Let Xl' ... , Xn be a random sample from the density I(x; 8), 8 E 8, where (3 IS some interval. Assume that I(x; 8) =
a(8)b(x) exp [c(8)d(x)], and set t(Xb ... , xn)

= L d(xJ
1 ..

If c(8) is a monotone, increasing function in 8 and if there exists k* such that P9o[t(X1, ... , Xn) > k*] = (x, then the test y* with a critical region C* = {(Xb ... , xn): t(Xl' . ", xn) > k*} is a uniformly most powerful size-a test of .Yf0: 8 <8 0 versus .Yf1 : 8 > 80 or of .Yf0: 8 = 80 versus .Yf1: 8 > 80 Oi) If c(8) is a monotone, decreasing function in 8 and if there exists k* such that P 9o [t(X1 , , Xn) < k*] = (x, then the test y* with a
(i)

COMPOSITE HYPOTHESES

423

critical region C* = {(Xl' ' , " x n): t(x l , " " Xn) < k*} is a uniformly most powerful size-a test of .Yt0: 0 00 versus J'l'l: 0 > 00 or of J'l' 0: 0 = 00 versus J'l'l: 0 > 00 , IIII EXAMPLE 13 Let Xl' , .. , Xn be a random sample from f(x; 0) = Oe- 8xI(0, oolx), where e = {O: 0 > O}. Test Jr 0: 0 < 00 '1ersus Jrl : 0 > 00 , f(x; 0) = 01(0. oo)(x) exp (-Ox) = a(O)b(x) exp [c(O)d(x)]; so t(xI , . , ' xn)

=I
I

Xb and c(O)

-0.

c(O) is a monotone, decreasing function; so

by (ii) of Theorem 5 a uniformly most powerful test is given by the following: Reject Jr0 if and only if I Xi < k*, where k* is given as a solution to

a = P 80 ['\' X I < k*] = '-'

n 8oU fr(n) on u - e- du
k

IIII

Definition 16 Monotone likelihood-ratio A family of densities {f(x; 0): oE 9, 9 an interval} is said to have a monotone likelihood-ratio
if there exists a statistic, say T = t(XI , ... , Xn), such that the ratio L(O'; Xl, ... , xn)/L(O"; X., .. ,' Xn) is either a nonincreasing function of t(x l , ... , xn) for every 0' < Olf or a nondecreasingfunction of t(x l , . , xn) for every Of < 0", IIII Note that in the term "monotone likelihood-ratio" the likelihood-ratio is not a generalized likelihood-ratio; it is a ratio of two likelihood functions. EXAMPLE 14 If {f(x; 0): 0 E 8} = {Oe- 8xI(0. oo)(x): 0 > O}, then
L(O'; x., ... , xn) L(O"; x., ... , xn)

= (O')n exp (- 0' I

(Ollr exp ( - 0" I

X,) = (o,)n exp [ _ (0' _ 0") I x.] Xi) 0" I

which is a monotone, increasing function in EXAMPLE 15 If{f(x; 0): OE9}

L(O'; x., ... , xn) L(O";
Xl, .. ,

Xi'

IIII

{(110)/(0.8)(x): 0 >O}, then

n

(1/8')"iU 1(0,8 )(xi) =(l/0't1(0,8/)(Yn)

Xn)

(l/O"yn 1(0,8") (Xi)

i= I

(l/0,,)nl(0,8")(Yn)

for

0 < Yn < 0'

for 0'::;; Yn < 0",

424

TBSTS OF HYPOTHESES

IX

which is a monotone, nonincreasing function in Yn = max [x h ... , xnl. [Note that YlI cannot fall outside of the interval (0, 0") when 0 is either 0' or 0".] IIII

Theorem 6 Let Xl"", Xli be a random sample fromf(x; 0), where e is

some interval. Assume that the family of densities {f(x; 0): 0 E t)} has a monotone likelihood-ratio in the statistic t(Xl , .. " Xn):
(i) If the monotone likelihood-ratio is nondecreasing in t(Xl, .. , xn) and if k* is such that P6o [t(Xl , ... , Xli) < k*] = Ci, then the test corresponding to the critical region C* = {(x h " ' , xn): t(Xl' ... , xn) < k*} is a uniformly most powerful test of size ct of J'l' 0: 0 < 00 versus J'l'1: 0 > 00 , (ii) If the monotone likelihood-ratio is noruncreasing in t(XI' .. , xn) and if k* is such that P"o[t(XI , ... , Xn) > k*] = a, then the test corresponding to the critical region C*={(xl , ... , xn): t(XI' . " Xli) > k*} is a uniformly most powerful test of size ct of J'l' 0: 0 < 00

versus J'l'l: 0 > 00

IIII

EXAMPLE 16 Let Xl, ... , Xn be a random sample from f(x; 0) = (110)1(0. 6)(X), where 0 > O. Test J'l'0: 0 00 versus J'l'1: 0 > 00 . We saw in Example 15 that the family of densities has a monotone, nonincreasing likelihood-ratio in t(Xh ... , xn) = YlI = max [Xh ... ,Xli]' According to Oi) of Theorem 6, a uniformly most powerful size-a test is given by the following: Reject J'l' 0 if YlI > k*, where k* is given as the solution to
ct

= P6o [Yll > k*]

dY fk n 0Y ) e;;
60

n- I

O~ [O~ -

(k*)lI]

= 1-

(k*) 11 0 ' 0

which implies that k* = 00 ViI - ct.

IIII

Several comments are in order. First, the null hypothesis was stated as o < 00 in both Theorems 5 and 6; if it had been stated as 0 > 00 , tIte two theorems would remain valid provided the inequalities that define the critical regions were reversed. Second, Theorem 5 is a consequence of Theorem 6. Third, the theorems consider only one-sided hypotheses. This completes our brief study of uniformly most powerful tests. We have seen that a uniformly most powerful test exists for one-sided hypotheses if the density sampled from has a monotone likelihood-ratio in some statistic. There are many hypothesis-testing problems for which no uniformly most powerful

COMPOSITE HYPOTHESES

425

test exists. One method of restricting the class of tests, with the hope and intention of finding an optimum test within the restricted class, is to consider unbiasedness of tests, to be defined in the next subsection.

3.3

Unbiased Tests

There are many hypotheses-testing problems for which a uniformly most powerful test does not exist. In these cases it may be possible to restrict the class of tests and find a uniformly most powerful test in the restricted class. One such class that has some merit is the class of unbiased tests.

Definition 17 Unbiased tests A test Y of the null hypothesis :Yf 0: (J E 8 0 against the alternative hypothesis :Yf1 : (J E e 1 is an unbiased test if and only if sup ny(J) < inf ny(J). I111
Be ~o Be ~1

Consequently in an unbiased test the probability of rejecting :Yt 0 when it is false is at least as large as the probability of rejecting :Yf 0 when it is true. In many respects this seems to be a reasonable restriction to place on a test. If within this restricted class a test exists that is uniformly most powerful, then we have a uniformly most powerfUl unbiased test. An elaborate theory has been developed for finding uniformly most powerful unbiased tests, but we will not study it. See [16].

3.4 Methods of Finding Tests

We presented in Subsec. 3.1 the generalized likelihood-ratio principle; it provides us with one method of obtaining tests of hypotheses. In Subsec. 3.2 we gave a method of finding a uniformly most powerful test for certain testing problems. There are still other methods of finding tests. One method is sketched on pages 456 to 459 of Subsec. 5.4 of this chapter. Another method, which might be called the confidence-interval method, is to use a confidence interval to obtain a test. F or instance, if it is desired to test :Yf0: (J = (J 0 versus :Yt1: (J =1= (J 0, then we might compute a confidence-interval estimate of (J from the data, and if the interval contains (Jo, accept :Yf0, and otherwise, reject it. If the confidence interval had a confidence coefficient y, then the resulting test would have size 1 - y. We will say more about how confidence intervals might be used to obtain tests in Sec. 6 below. A useful and intuitive technique for obtaining tests is the following: Discover some statistic which behaves differently under the two hypotheses, and

426

TESTS OF HYPOTHESES

IX

utilize the different behavior to design a test. As an illustration, consider testing .Yf0: e ~ eo versus Jr 1 : e > eo, where the sample Xl, ... , Xn is selected from the density I(x; e) = 4>8,1 (x). The statistic X has a normal distribution with mean e and variance 1In; hence the statistic X will tend to be smaller when .Yf0 is true than when .Yf0 is false. The statistic X behaves differently under the two hypotheses. A reasonable test, then, would be to reject .Yf0 for X large; that is, reject .Yf 0 if X > k, where k is determined by, say, fixing the size of the test. (We know from Subsec. 3.2 above that such a test is uniformly most powerful.) To employ this technique, a statistic has to be discovered which will behave differently under the two hypotheses. There are various ways of approaching the task of discovering such a statistic. For instance, if a sufficient statistic exists, then it is a natural candidate to try; or a good estimator, such as a maximum-likelihood estimator, of the parameter or parameters that are used to specify the hypotheses is another possibility for the needed statistic. In the above simple illustration X was all these since X is the maximum-likelihood estimator of e as well as being a sufficient statistic. We make frequent use of this intuitive technique for obtaining tests in the remaining sections. EXAMPLE 17 Let Xl, ... , Xn be a random sample from a Poisson distribution with mean e. Suppose that it is desired to test that the mean is a fixed value, say eo; that is, test .Yf 0: e = eo versus .Yf1 : e =/; eo. We know that X is the maximum-likelihood estimator of e and that X will tend to be distributed about eo if .Yf0 is true. Consequently, the following test seems reasonable: Accept.Yf 0 if C1 < x < C2' and otherwise reject it, where C1 and C2 are selected so that the test will have a desired size. To be specific, let n = 10 and eo = I. The test given by "Accept .Yf 0 if and only if.4 < x < 1.6" has size given by
I - P8=1[.4 < X < 1.6] = I - P8=1[4 < Xi < 16] 15 e- 10 loi =I~ .078~ 5 j!

IIII

A test that has been quite extensively applied in various fields of science is .Yf0: e = eo against .Yf1 : e =/; eo. For example, let e be the mean difference of yields between two varieties of wheat. It is often suggested that it is desirable to test the hypothesis .Yf0: e = 0 against .Yf1 ; e =/; 0, that is, to test if the two varieties are different in their mean yields. However, in this situation, and many others where e can vary continuously in some interval, it is inconceivable that e is exactly equal to 0 (that the varieties are identical in their mean yields). Yet this is what the test is stating: Are the two mean yields identical (to one

COMPOSITE HYPOTHESES

427

ten-billionth of a bushel, etc.)? In many cases it seems more realistic for an experimenter to select an interval about 00 , say 01 < 00 < O 2 , and test :Yf0: 01 o< O2 against the alternative :Yf1 : 0 < 01 or 0 > O2 For example, it may be feasible to set 01 = -j- and O 2 = t in the above illustration and test if the difference of the mean yields of the two varieties is between -t bushel and +1 bushel against the alternative that it is not in this interval. A test that is uniformly most powerful for the above hypothesis may be difficult or impossible to devise, but if/ex; 0) is a density with a single parameter, then the maximum likelihood estimator (3 may sometimes be used to construct a test and the power of this test compared with the ideal power function for a test of size ct. A test of the following form may be used for some densities: Reject lito if () is not in some interval, say (Ch C2), and accept :Yf 0 if t} is in the interval, where C1 and C2 are chosen so that the test has size ct. Often Cl and C2 can be chosen so that

where/~(t}; 0) is the density of (3 when 0 is the parameter.

The power function

of this test is

nCO)

1-

C2

f ~(t); 0) dt}

for 0 in 9.

Cl

This power function can be compared with the ideal power function, and if it does not deviate further from the ideal than the experimenter can tolerate, the test may be useful even though it may not be a uniformly most powerful test. Let us illustrate the above with a simple example. EXAMPLE 18 Let Xl' ... , Xn be a random sample from 4>(J,l(X). Test :Yf 0: 1 <0 < 2 versus :Yf1 : () < I or 0 > 2. X is the maximum-likelihood estimator of 0; it has a normal distribution with mean 0 and variance lin. According to the above we would like to select Cl and C2 so that

We have

~(~) - ~(J~) = ~(JD-t) - ~(~) =

and we can see from Fig. 5 that C1 by, say,
=

1-

~,

i -

d and

C2

i + d, where d is given

<ll ( d

Jl/n

t) -<I> (~~ -I-ct Nn .

2

428

TESTS OF HYPOTHESES

IX

.5

FIGURE 6

.05 ~_ _----'..::::_"""~L...-...__--'---_---+ e o 1 2 3

For example, if ex = .05 and n = 16, then d -:: : ; .911; so C2 -:::::; 2.411. The power function is given by
nCO)

Cl -:::::;

.589, and

= 1 - Pe[cl < X < c2l = 1 - Pe[.589 < X < 2.411]

and is sketched in Fig. 6.

IIII

TESTS OF HYPOTHESES-SAMPLING FROM THE NORMAL DISTRIBUTION

A number of the foregoing ideas are well illustrated by common practical testing problems-those problems of testing hypotheses concerning the parameters of normal distributions. The section is subdivided into four subsections, the first two dealing with just one normal population and the last two dealing with several normal populations.

4.1

Tests on the Mean

We shall assume that we have a random sample of n observations Xl' ... , Xn from a normal population with mean J.l and variance (J2, and we will be interested in testing hypotheses about J.l. There is quite a variety of hypotheses about the mean J.l that can be formulated; we begin by considering one-sided hypotheses.

TESTS OF HYPOTHESES-SAMPLING FROM THE NORMAL DISTRIBUTION

429

:Yt 0: Jl < Jlo versus :Yt I : Jl > Jlo In testing :Yt0 : Jl S Jlo versus :Yt I : Jl > Jlo there are two cases to consider depending on whether or not (]2 is assumed known, If (]2 is assumed known, our parameter space is the real line, and we are testing a one-sided hypothesis; so we have hope of finding a uniformly most powerful test, Since (]2 is assumed known, it is a known constant; hence

1 J2n(]

e-HIl/tI)2e-!(x/tI)2e(lt/tI)X,

which is a member of the exponential family with

( ) aJl_
k e y2n(]

-Hllltl)2

and

d(x) = x.

The conditions for Theorem 5 are satisfied; so the uniformly most powerful size
ct

test is given by the following: Reject:Yt 0 if t(XI' .,., Xn) = I

n Xi

> k~', where k*

is given as a solution to Pllo[l: Xi > k*] = ct. Now ct = Plto [I Xi > k*] = I - q,k* - nJlo)/jn(J); so (k* - nJlo)/jn(] = ZI-a' where Zl-a is the (1 - ct)th quantile of the standard normal distribution, The test becomes the following: Reject :Yt 0 if I Xi > nJlo + In(]ZI-a, or reject :Yt 0 if x >- Jlo + (J/jn)zl_a' If (J2 is assumed unknown, then testing :Yt0: Jl ~ Jlo versus :Yt1 : Jl > Jlo is equivalent to testing :Yt0: (J E 9 0 versus :Yt1 : (J ~ where (J = (Jl, (]2), 9 = {(Jl, (J2): - 00 < Jl < 00; (J2 > O}, and 9 0 = {(Jl, (]2): Jl ~ Jlo; (J2 > O}. Toobtain a test, we could use the generalized likelihood-ratio principle, or we could find some statistic that behaves differently under the two hypotheses and base our

eo,

test on it. Such a statistic is T = (X - Jlo)/(Sljn), where X is the sample mean and S2 is the sample variance, Since Twould tend to be larger for Jl > Jlo than for Jl < Jlo , a test based on T is given by the following: Reject :Yt 0 if T is large; that is, reject :Yt0 if T> k, If Jl = Jlo, then T has a t distribution with n - I degrees of freedom; so k can be determined by setting ct = PIl =Ilo[T > k], which implies that k = tl-a:(n - I), the (1 - ct)th quantile of a t distribution with n - I degrees of freedom, It can be shown that the test derived here is a generalized likelihoodratio test having size ct.
ft' 0: J1.

= J1.o versus :YtI : Jl =1= Jlo Again, we have two cases to consider dependFor (Jz known, we know that

ing on whether or not (Jz is assumed known. (X - z(l +"f)/2(JI.j~), X

+ Z(l +"f)/z(Jljn)) is a lOOy percent confidence interval

430

TESTS OF ID'PQTHESES

IX

for Jl, where Z(l + y)/2 is the [(1 + y)/2]th quantile of the standard normal distribution. A possible test is given by the following: Reject .Yt0 if the confidence interval does not contain Jlo . Such a test has size I - y since
Pp.=p.o [ X

- Z(1+y)/2

(J (J ] J;';' < Jlo < X + Z(1+y)/2 yin = y.

If (12 is assumed unknown, we could obtain a test, similar to the one above, using the lOOy percent confidence interval

Instead, let us find a generalized likelihood-ratio test.

L(P.

(12; Xl>""

X.) =

(~J exp [ - ~

Ie' P)1
~
nl2
e

9 0 = {(Jl, (J2): Jl = Jlo; (12 > O}, and 8 = {(Jl, (J2): - 00 < Jl < 00; (12 > O}. We have already seen that the values of Jl and (12 which maximize L(Jl, (12; Xl' ... , xn) in 8 are fl = x and fJ2 = (lIn) (Xi - x)2; so

sup L(Jl,
~

2.

(1

,Xb , Xn) - [ 2 ~ _2 n ~ (Xi - x) ]

-n12

To maximize Lover 8 0 , we put Jl = Jlo, and the only remaining parameter is (12; the value of (12 which then maximizes L is readily found to be (12 = (lIn) L (Xi - Jlo)2, which gives sup L ( Jl,
~o
2.
(1

,Xl"'" Xn) -

2n

~ 2 ~ (Xi - Jlo)

] nl2 _ nl2
e

The generalized likelihood-ratio is then

We note now that A is a monotonic function of

TESTS OF HYPOTHESES-SAMPLING FROM tHE NORMAL DISTRIBUTION

431

and so a critical region of the form A < Ao is equivalent to a critical region of the fonn t2(Xl' ... , x n ) > k 2. A generalized likelihood-ratio test is then given by the following: Reject :Yf 0 if and only if

T2=

[~/lnr >k2,

or accept :1'f 0 if and only if - k < T < k. Since T has a t distribution with n - I degrees of freedom when J-l = J-lo' if k is selected SO that J~k fT(t; n - 1) dt = 1 - a, then our test will have size a. k is given by tl -a/2(n - 1), the (l - a/2)th quantile of a t distribution with n - 1 degrees of freedom. We might note that this size-a test obtained by using the generalized likelihood-ratio principle is the same size-a test that we obtained above using the confidence-interval method of obtaining tests with the confidence interval

_ (X -

t(1+I')/2(n - 1)

In' X + t(1+I')/in -

s _

1)

In '

s)

where y = I - a. Although we will not prove it, the test that we have obtained is uniformly most powerful unbiased. We have found tests on the mean of a normal distribution for both onesided and two-sided hypotheses. One might note that the one-sided null hypothesis J-l < J-lo could be reversed and comparable results obtained. There are other hypotheses about the mean that could be formulated, such as :Yf 0: J-ll < J-l < J-l2 versus :Yf1: J-l < J-ll or J-l > J-l2 .

4.2

Tests on the Variance

As in the last subsection, we shall assume that we have a random sample of size n from a normal population with mean J-l and variance a 2. We will be interested in testing hypotheses about a 2
:Yf 0: a < a~ versus :Yf 1 : a 2 > a~ There are two cases to consider depending on whether or not J-l is assumed known. If J-l is known, then our parameter space is an interval, and our hypothesis is one-sided; so we have a chance of finding a uniformly most powerful size-a test.
2

f(x; 8)

f(x; ( 2 )

1 fi
2na

e-(1/2a )(x-Il)2,

which is a member of the exponential family with a(a 2 ) = (2m'r 2 )-t, b(x) = 1, 2 c(a ) = - 1/2a2, and d(x) = (x - J-l)2. [J-l is known; so d(x) is a function of x 2 only] c(a ) is a monotone, increasing function in a 2 ; so, by Theorem 5, the

432

TESTS OF

HYPOTHESES

IX

test with critical region = {(Xl' ... , Xn): L (Xi - 1l)2 > k*} is uniformly most powerful of size ct, where k* is given by Pa2=ao2[L (Xi - 1l)2 > k*] = ct, which implies that k* = (}"~ xi -in), where xi -cz(n) is the (l - ct)th quantile point of the chi-square distribution with n degrees of freedom. If Il is unknown, a test can be found using the statistic V = L (Xi - X)2/(}"~. V will tend to be larger for (}"2 > (}"~ than for (}"2 < (}"~; so a reasonable test would be to reject :Yf 0 for V large. If (}"2 = (}"~ , then V has a chi-square distribution with n - I degrees of freedom, and Pa2=ao2[V> xi-in - I)] = ct, where xi -cz(n - I) is the (l - ct)th quantile of a chi-square distribution with n - I degrees of freedom. It can be shown that the test given by the following: Reject :Yfo if and only if L (Xi - X)2/(}"~ > xi-cz(n - I) is a generalized likelihood-ratio test of size ct.

versus :Yfl : (}"2 =1= (}"~ We leave the case Il assumed known as an exercise. For Il unknown, so that 8 0 = {(Il, (}"): - 00 < Il < 00; (}"2 = (}"~}, we can find a size-ct test using the confidence-interval method. In Subsec. 3.2 of Chap. VIII, we found the following lOOy percent confidence interval for (}"2:
:Yf 0:
(}"2

= (}"~

(n - I)S2, (n - l)S2), (
q2 ql

where ql and q2 are quantile points of a chi-square distribution with n - I degrees of freedom, say fQ(q; n - I), satisfying -.

A size-(ct = I - y) test is given by the following: Accept :Yf 0 if and only if (}"5 is contained in the above confidence interval. It is left as an exercise to show that for a particular pair of ql and q2 the test of size ct derived by the confidenceinterval method is in fact the generalized likelihood-ratio test of size ct.

4.3

Tests on Several Means

In this subsection we will consider testing hypotheses regarding the means of two or more normal populations. We begin with a test of the equality of two means.

Equality of two means In many situations it is necessary to compare two means when neither is known. If, for example, one wished to compare two proposed new processes for manufacturing light bulbs, one would have to base the comparison on estimates of both process means. In comparing the yield of

TESTS OF HYPOTHESE8-SAMPLING FROM THE NORMAL DISTRIBUTION

433

a new line of hybrid com with that of a standard line, one would also have to use estimates of both mean yields because it is impossible to state the mean yield of the standard line for the given weather conditions under which the new line would be grown. I t is necessary to compare the two lines by planting them in the same season and on the same soil type and thereby obtain estimates of the mean yields for both lines under similar conditions. Of course the comparison is thus specialized; a complete comparison of the two lines would require tests over a period of years on a variety of soil types. The general problem is this: We have two normal populations-one with a random variable Xl' which has a mean III and variance CTi, and the other with a random variable X 2 , which has a mean 112 and variance CTi. On the basis of two samples, one from each population, we wish to test the null hypothesis versus

.Yt 1: III # 112 , CTi > 0, CTi > O.

The parameter space 9 here is four-dimensional; a joint distribution of Xl and X 2 is specified when values are assigned to the four quantities (Ill' 1l2, CTi, CTD. The subspace 9 0 is three-dimensional because values for only three quantities (Il, CTi, CT~) need be specified in order to specify completely the joint distribution under the hypothesis that III = 112 = Il, say. We shall suppose that there are nl observations (X11' X 12 , ... , X 1n1 ) in the sample from the first population and n2 observations (X21' X 22 , ... , X 2n ) from the second. The likelihood function is

_ (-12)nl/2 exp
2nCTl

[l I
-"!

nl (Xli-lll)2]( - 12 )n2/2 exp [ -t I n2 (X2J-1l2)2] 1 CTl 2nCT2 1 CT2

'

and its maximum in 9 is readily seen to be

Ifwe put III and 112 equal to Il, say, and try to maximize L with respect to Il, CTi, and CT~, it will be found that the estimate of Il is given as the root of a cubic equation and will be a very complex function of the observations. The resulting generalized likelihood-ratio A will therefore be a complicated function, and to find its distribution is a tedious task indeed and involves the ratio of the two variances. This makes it impossible to determine a critical region 0 < A < k

434

TISTS OF BYPOTHBSES

IX

for a given probability of a Type I error because the ratio of the population variances is assumed unknown. A number of special devices can be employed in an attempt to circumvent this difficulty, but we shall not pursue the problem further here. For large samples the following criterion may be used: The root of the cubic equation can be computed in any instance by numerical methods, and A can then be calculated; furthermore, as we shall see in Sec. 5 below, the quantity - 2 log A has approximately the chisquare distribution with one degree of freedom, and hence a test that would reject for - 2 log )" large could be devised. When it can be assumed that the two populations have the same variance, the problem becomes relatively simple. The parameter space 9 is then three .. dimensional with coordinates (Ilh Ilz, (1z), while 9 0 for the null hypothesis III = Ilz = Il is two-dimensional with coordinates {Jl, (1z). In 9 we find that the maximum-likelihood estimates of Ill' Ilz, and (1z are, respectively, Xf, Xz, and

so

for Il and

which gives supL

~o

nl [ 2n

+ n2

](n +n )/Z
1 2

[I

(Xli - XI)Z

(XZi -

x )Z +
2

nlnz (Xl - Xz)z] nl + n2

TESTS OF HYPOTHESES-SAMPLING FROM THE NORMAL DISTRIBUTION

435

Finally,

it

(1 +

[n l n2/(n l ~ n2)](XI (Xli -

X2)2

Xl)

+ I(X2j - x2 )

2)

2 -(n1 +n )/2.

(17)

This last expression is very similar to the corresponding one obtained in Subsec. 4.1, and it turns out that this test can also be performed in terms of a quantity which has the t distribution. We know that Xl and X 2 are independently normally distributed with means III and 112 and with variances (i /n l and u 2 /n2' Also it is readily seen that Xl - X 2 is normally distributed with mean III - 112 and variance u 2(1/nl + I/nz) Under the null hypothesis the mean of Xl - X z will be O. The quantities I (XIt - XI)2ju 2 and I (XZ) - Xz)Z/u 2 are independently distributed as chi-square distributions with nl - I and n z - I degrees of freedom, respectively; hence their sum has the chi-square distribution with nl + nz - 2 degrees of freedom. Since under the null hypothesis
Z=

~l-XZ
uJI/nl

+ l/nz
(18)

is normally distributed with mean 0 and unit variance, the quantity

T=

J[I

JnlnZ/(nl + n2)(XI - X 2) (Xli - XI)z + I (XZi - X 2)Z]/(nl

+ nz -

2)

has the t distribution with nl + nz - 2 degrees of freedom. [Note that we do have independence of the numerator and denominator in Eq. (18).] The generalized likelihood-ratio is
(19)

and its distribution is determined by the t distribution. The test would, of course, be done in terms of T rather than 2. A 5 percent critical region for Tis Z T > [t.97S(nl + nz - 2)]2, where l.97S(nl + nz - 2) is the .975th quantile of the t distribution with nl + n2 - 2 degrees of freedom. If we want to test :Yf 0: III = 112 versus :Yf I : III > Ilz or :Yf 0: III ::;;; Ilz versus :Yf I: III > Ilz, a size-IX test is given by the following: Reject :Yf 0 if and only if T> tl-lnl + n z - 2), where Tis defined in Eq. (18) and tl-a:(n l + n2 - 2) is the (1 - IX)th quantile of the t distribution with nl + nz - 2 degrees of freedom.
Equality of several means The test presented above can be extended from just two normal popUlations to k normal populations. We assume that we have available k random samples, one from each of k normal populations; that is,

436

TBSTS OF HYPOTHESES

IX

let Xii' .. , XJnJ be a random sample of size nj from the jth normal population, j = I, ... , k. Assume that the jth population has mean Ii) and variance (12. Further assume that the k random samples are independent. Our object is to test the null hypothesis that all the population means are the same versus the alternative that not all the means are equal. We seek a generalized likelihood ratio test. The likelihood function is given by

where n = The

L
j= I

nj.

parameter space 9 is (k + 1)-dimensional with coordinates (iii' ... , lik' (12), and 9 0 , the collection of points in the parameter space corresponding to the null hypothesis, is two-dimensional with coordinates (Ii, (12), where Ii = iii = ... = lik' In 9, the maximum-likelihood estimates of iii' ... , lik, (12 are given by
I nJ
r
A II j =X. J.

=-

n)l= I

x", JI

= 1, ... , k,

and
1
"'2 = (1~

n j=l

k L I( x .. - x - )2.
HJ

i=l

)1

J.'

(20)

hence,

In 9 0 , the maximum-likelihood estimates of Ii and and and so

21l ~ ~ (Xji

(12

are

sup L =
o

)~

_X)2] -n12

e- nI2

TESTS OF HYPOTHESES-SAMPLING FROM THE NORMAL DISTRIBUTION

437

~~

(Xji -

X J,)

1+

k - 1 ~ nix), - x)2/(k - 1)
n - k ~~(Xjl- x),) I(n - k)
J

l-n'2

A generalized likelihood-ratio test is given by the following: Reject :Yf 0 if and only if A Ao ' But A < Ao if and only if

niXj. - x)2j(k - 1)
2

r= " "

j= 1

~~ (Xji - XJ.) I(n - k) J i

> some constant, say c.

(21)

The ratio r is sometimes called the variance ratio, or F ratio. The constant c is determined so that the test will have size (X; that is, c is selected so that P[R > c 1:Yf0] = (x. Note that Xj. is independent of I (Xji - X l )2 and,hence, the
i

numerator of Eq. (21) is independent of the denominator. Also, under :Yf 0' note that the numerator divided by (12 has a chi-square distribution with k - I degrees of freedom, and the denominator divided by (12 has a chi-square distribution with n - k degrees of freedom. Consequently, if :Yf 0 is true, R has an F distribution with k - land n - k degrees of freedom; so the constant c is the (1 - (X)th quantile of the F distribution with k - 1 and n - k degrees of freedom. The testing problem considered above is often referred to as a one-way analysis of variance. In some experimental situations, an experimenter is interested in determining whether or not various possible treatments affect the yield. F or example, one might be interested in finding out whether various types of fertilizer applications affect the yield of a certain crop. The different treatments correspond to the different populations, and when we test that there is no population difference, we are testing that there is no treatment" effect. The term " analysis of variance" is explained if we note that the denominator of the ratio in Eq. (21) is an estimate of the variation within populations and the
U

 ..
,

,.-

438

TESTS OF HYPOTHESES

IX

numerator is an estimate of the variation between populations when means are equal. We are analyzing variance to test equality of means. 4.4 Tests on Several Variances

Two variances Given random samples from each of two normal populations with means and variances (/11' uI) and (/12' ui), we may test hypotheses about the two variances. We will consider testing: (i) (ii) (iii)

ui < ui versus .Yf1: ui > ui .Yf0: ui > ui versus .Yf1: ui < ui .Yf0: ui = ui versus .Yf1: ui =1= ui
.Yf0:

If X 11 , .. , X l711 is a random sample from a normal density with mean /11 and variance ui, if X 21 , .. , X 2712 is a random sample from a normal density with mean /12 and variance ui, and if the two samples are independent, then we know that

has the F distribution with n1 ular, the statistic

1 and n2 - 1 degrees of freedom, and, in partic-

has the F distribution with nl - 1 and n2 - 1 degrees of freedom when ui = ui . Note that the statistic R tends to be large when ui > ui and small when ui < ui, and so we can capitalize on this different behavior to formulate tests for the hypotheses (i) to (iii). F or instance, in testing .Yf0: ui < u 2 versus .Yf1: ui > ui, we would reject .Yf0 for large R, or a size-a test is given by the following: Reject .Yf 0 if and only if R exceeds F1 - rz (n 1 - 1, n 2 - 1), the (1 - a)th quantile of the F distribution with nl - 1 and n 2 - 1 degrees of freedom. Similarly, a test of .Yf0: ui > ui versus .Yf1: ui < ui is given by the following: Reject .Yf0 if and only if R is less than Finl - 1, n2 - 1), the ath quantile of the F distribution with n 1 - 1 and n2 - 1 degrees of freedom. A test of .Yf0: ui = ui versus .Yf1 : ui =1= ui should be two-tailed; that is, .Yf0 should be rejected for small or large R. In other words, a test is given by the following: Accept .Yf0 if and only if kl < R < k2' where kl and k2 are selected so that the test will have size a. It is customary to make the two tails have equal areas of al2 (although this is not quite the best test); then kl = Frz / 2(n 1 - 1, n2 - 1), and k2 =
F1 rz / 2 (n 1 -

1, n 2

1).

TEST OF HYPOTHESES--SAMPUNG FROM THE NORMAL DISTRIBUTION

439

We might mention that the above defined tests can all be derived using the generalized likelihood-ratio principle. Equality of several variances Let X jI , .. , Xj"j be a random sample of size nj from a norma] population with mean Jl j and variance a}, j = 1, ... ,k. Assume that the k samples are independent. Our object is to test the null hypothesis :If0: (1'I = (1'i = ... = (1'~ against the alternative that not all variances are equal. The likelihood function

- n n .1
k"j
j= 1

i~ 1

J21l (1'j
=

e- U (XJI-Jl.J)/O'jJ2
1, ... , k are given by

and the maximum-likelihood estimates of Jlj' (1'1, j

and

hJ = - I

nj

nJ i= 1

(Xji - XjJ z.

The null hypothesis states that all (1'] are equal. Let (1'2 denote their common value; then So = {(Jll' ... , Jlk' (1'2): - 00 < Jlj < 00; (1'2 > O}, and the maximum likelihood estimates of JlI" .. , Jlk' (1'2 over So are given by
j

= 1, ... , k,

and

Therefore,

) _
,-

s~f L
s~p L _

(fzt'J12 exp (-2/2)

Ii (~)"j/2 exp (- I
j= 1

nj/2)

(})

j=1

n (hJ)"i/
k

- (I n) hJ/I n)Y'''j/z'
A generalized likelihood-ratio test is given by the following: Reject :If 0 if and only if A :S; Ao . We would like to determine the size of the test for any constant

440

TESTS OF HYPOTHESES

IX

AO or find Ao so that the test has size a, but, unfortunately, the distribution of the
generalized likelihood-ratio is intractable. An approximate size-a test can be obtained for large nJ since it can be proved that - 2 log A is approximately distributed as a chi-square distribution with k - 1 degrees of freedom. Accord~ ing to the generalized likelihood~ratio principle .Yf0 is to be rejected for small A; hence .Yf 0 should be rejected here for large - 2 log A; that is, the critical region of the approximate test should be the right tail. So the approximate size-a test is the following: Reject .Yf 0 if and only if - 2 log A > Xl -a.Ck - I), the (l - a)th quantile of the chi~square distribution with k - I degrees of freedom. (Several other approximations to the distribution of the likelihood~ratio statistic have been given, and some exact tests are also available.)

5 CHI-SQU ARE TESTS

In this section we present a number of tests of hypotheses that one way or another involve the chi-square distribution. Included will be the asymptotic distribution of the generalized likelihood-ratio, goodness-of-fit tests, and tests concerning contingency tables. The material in this section will be presented with an aim of merely finding tests of certain hypotheses, and it will not be presented in such a way that concern is given to the optimality of the test. Thus, the power functions of the derived tests will not be discussed.

5.1

Asymptotic Distribution of Generalized Likelihood-ratio

On two occasions in Sec. 4 we found that the distribution of the generalized likelihood-ratio was intractable, and both times we indicated that an approximate test could be obtained by using an asymptotic distribution of the generalized likelihood-ratio. The following theorem, which we shall not be able to prove because of the advanced character of its proof, gives the asymptotic distribution of the generalized likelihood~ratio.

Theorem 7 Let X b , Xn be a sample with joint density Ix., .. , Xn (', ... , . ; 0), where 0 = (0 1 , . , Ok)' that is assumed to satisfy quite general regularity conditions. Suppose that the parameter space (3 is k-dimen~ sional. In testing the hypothesis

em-SQUARE TESTS

441

where O~, .. , O~ are known and Or+l' .. , Ok are left unspecified, -2 log An is approximately distributed as a chi-square distribution with r degrees of freedom when :If 0 is true and the sample size n is large. //11 We have assumed that 1 <r < k in the above theorem. If r = k, then all parameters are specified and none is left unspecified. The parameter space e is k-dimensional, and since :If 0 specifies the value of r of the components of (01 , . , Ok)' the dimension of So is k - r. Thus, the degrees of freedom of the asymptotic chi-square distribution in Theorem 7 can be thought of in two ways: first, as the number of parameters specified by :If 0 and, second, as the difference in the dimensions of and 9 0 , Recall that An is the random variable which has values

which in tum is the generalized likelihood-ratio for a sample of size n. 9 0 is that subset of 9 that is specified by :If o. The generalized likelihood~ratioprin~ ciple dictates that :If 0 is to be rejected for A" small, but since - 2 log An increases as An decreases, a test that is equivalent to a generalized likelihood-ratio test is one that rejects for -2 log A" large. Now, since the theorem gives an approximate distribution for the values - 2 log An when :If 0 is true, a test with approximate size a; is given by the following: Reject :If 0 if and only if - 2 log An >

xf _(r),

where XI _(r) is the (l - a)th quantile of the chi-square distribution with r degrees of freedom. Note that the degrees of freedom r is the number of components of the parameter space that are specified by the null hypothesis. Because of the specific form of the null hypothesis in the theorem, it may appear that the result is not too widely applicable. The null hypothesis of the theorem specifies the values of a subset of the k components of the k-dimensional parameter space, and not many null hypotheses are of that form. However, often the density can be reparameterized so that the null hypothesis is of the form given in the theorem. We illustrate with two examples.

EXAMPLE 19 Recall that in Subsec. 4.3 we discussed testing :If 0: III 1l2' O'f > 0, O'i > 0 versus :lfl : III =1= 1l2' O'i > 0, O'i > 0, where III and are the mean and variance of one normal population and 112 and O'i are the mean and variance of another. Here the parameter space is fourdimensional, and although :If0 does not appear to be of the form given

0';

442

TESTS OF HYPO'IHESES

IX

in Theorem 7, we can reparameterize to make it of that form. Let 01 = Jl1 - 112, O2 = Jl2' 03 = O'r, and 04 = O'~ In terms of the reparameterization, :Yf 0 becomes :Yf 0: 01 = O~ = 0, O 2 , 03 , 04 ; that is, the component 01 is specified to be 0, and the remaining three components are unspecified. The theorem is now applicable for the reparameterization; that is, the asymptotic distribution of - 2 log A' is known (and is the chisquare distribution with one degree of freedom) for :Yf 0 true, where A' is the generalized likelihood-ratio obtained under the reparameterization. However, because of the invariance property of maximum-likelihood estimators, the generalized likelihood-ratio A' obtained under the reparameterization is the same as the generalized likelihood-ratio A obtained before reparameterization. jill

EXAMPLE 20 In Subsec. 4.4 we tested :Yf 0: O'r = ... = 0';, 111' ... , 11m, where I1j and 0'] were, respectively, the mean and variance of the jth normal population, j = 1, ... , m. (In Subsec. 4.4, k was used instead of m.) If we make the following reparameterization, :Yf 0 will have the desired form of Theorem 7:

Now:Yf o becomes :Yfo: 01 = 1, ... , 0m-l = 1, Om, Om+l'"'' 02m; that is, the first m - 1 components are specified to be 1 and the remaining are unspecified. Theorem 7 is now applicable, and, again, because of the invariance property of maximum-likelihood estimates, the generalized likelihood-ratio obtained before and after reparameterization are the same; hence the asymptotic distribution of - 2 log A, as claimed in Subsec. 4.4, is the chi-square distribution with m - 1 degrees of freedom when :Yf 0 is true. // /I

5.2 Chi-square Goodness-of-fit Test

We commence this section with an example of a testing problem that involves the specification of the parameters of a multinomial distribution. It is hoped that this example will help motivate the presentation of the goodness.of-fit test. If a population has a multinomial density
"+1

f(x 1 ,

x" ; Ph "', p,,) =

J=l

n pji,

(23)

CHI-sQUARE nsTS

443

where xi = 0 or 1, j = 1, ... , k
k+l

+ 1; 0 <

Pj < 1, j = 1, "', k

+ 1; I

1+1
Xj

= 1;

j= 1

and

population of individuals who could be classified into k + I classes or categories), a common problem is that of testing whether the probabilities Pj have specified numerical values. Thus, for instance, the result of casting a die may be classified into one of six classes, and on the basis of a sample of observations we may wish to test whether the die is true, that is, whether Pj = t for j = 1, ... , 6. One can also think in terms of independent, repeated trials, where each trial can result in anyone of k + 1 outcomes, called classes or categories. The density in Eq. (23) then gives the density for the outcome of one trial. The result of one trial can be represented by the multivariate random variable (Xl' . , Xk), where Xj is unity if the trial results in category j and is 0 otherwise. Pj is the probability that a trial results in category j. Now if we independently repeat the trial n times, we have n observations of the multivariate random variable (Xl' ... , X k); we can display them as

j=l

Pj = 1 (as would be the case in sampling with replacement from a

If we let N j = I" Xl}' then the random variable N J is the number of the n trials
t= 1

resulting in category j. We know that (Nl' ... , N k ) has a multinomial distribu.. tion. (See Example 5 in Subsec. 2.2 of Chap. IV.) To test the null hypothesis .1fo: Pj = pJ, j = 1, ... , k + 1, where pJ are given probabilities summing to unity, we hope to employ the generalized likelihood-ratio principle. The likelihood function is given by
L = L(Pl' .. , Pk;
X 1b , Xu, . , X"h .. ,

X"k) =

i= 1 }= 1

" k+ n n pjli.
1

(24)

. The parameter space e has k dimensions (given k of the k + 1 P/s, the remaining one is determined by I Pj = 1), while 9 0 is a point. It is readily found thatL is maximized in 9 when

where

nj

is a value of the random variable N j

Hence, sup L
~

1 k+1

n J=1

n njJ.

444

TESTS OF HYI'01"HBSES

IX

The maximum of Lover likelihood-ratio is

eo is its only value j=l n (P1 )'tJ, and so the generalized

1;+1

A = n"

k+l (pO)"J

1'=1

-1
nj

A generalized likelihood-ratio test is given by the following: Reject JIP 0 if and only if A < )'0 , where the constant Ao is chosen to give the desired probability of a Type I error. For small n, the distribution of the generalized likelihood-ratio may be tabulated directly in order to determine Ao ; for large values of n, we may use Theorem 7, which states that -2 log A has approximately the chi-square distribution with k degrees of freedom. The chi-square approximation is sur~ prisingly good even if n is small provided that k > 2. Another test which is still commonly used for testing JIP 0 was proposed (by Karl Pearson) before the general theory of testing hypotheses was developed. This test uses the statistic
o k+l (N j Qk = I
j=1
-

npJ)2
0'

np1'

(25)

which tends to be small when JIP 0 is true and large when JIP 0 is false. Note that N j is the observed number of trial outcomes resulting in category j and nPJ is the expected number when JIP 0 is true. It can be easily shown (see Prob. 39) that k+l I
S[Q~] =
J= 1 np J

-0

[np/l - Pj)

+ n2 (pj -

pJ)2],

(26)

where the Pj are the true parameters. If JIP 0 is true, then S[Q~] = I (1 - pJ) = k + 1 - 1 = k. The following theorem gives a limiting distribution for Q~ when the null hypothesis .;t(? 0 is true. Let the possible outcomes of a certain random experiment be decomposed into k + 1 mutually exclusive sets, say Ab "', A k + 1 Define PJ = P[A j], j = I, ... , k + 1. In n independent repetitions of the random experiment, let N j denote the number of outcomes belonging to
Theorem 8

set A j , j

= 1, ... ,

+ 1, so that I

k+l

Nj

= n. Then
Qk =
k+l (N. - np.)2

j=1

(27)

j=1

npj

has as a limiting distribution, as n approaches infinity, the chi-square distribution with k degrees of freedom. 1//1

CHI-SQUARE TESfS

445

We will not prove the above theorem, but we will indicate its proof for k = 1. What needs to be demonstrated is that for each argument X~ FQk(x) converges to Fx2(k/x) as n --'). 00, where FQk(') is the cumulative distribution 2(k)(') is the cumulative distribution function of the random quantity Qk and F X function of a chi-square random variable having k degrees of freedom. (Note that k + 1, the number of groups, is held fixed, and n, the sample size, is increasing.) If k = 1, then

We know that NI has a binomial distribution with parameters n and PI and that Y,. = (Nl - npI)/JnpI(l - PI) has a limiting standard normal distribution; hence, since the square of a standard normal random variable has a chi-square = Q1 has a limiting distribution with one degree of freedom, we suspect that chi-square distribution with one degree offreedom, and such can be easily shown to be the case, which would give a proof of Theorem 8 for k = 1. Theorem 8 gives the limiting distribution for the statistic

Y;

(N. _ npO)2 o _ k+1 ~ J j Qk-.i..J }=1 npj

when the null hypothesis Yf o : p) = pJ,j = 1, ... , k + 1, is true. Thus a test of Yf 0: p} = pJ, j = 1, ... , k + 1, which has approximate size a, is given by the following: Reject Yf if and only if Q~ >

xr -a.(k),

the (1 - a)th quantile of the chi-square distribution with k degrees of freedom. We now have two large-sample tests of the null hypothesis Yf o : Pj = pJ, i = 1, ... , k + I, the one just defined, which uses Theorem 8, and the other given in terms of the generalized likelihood-ratio, which uses Theorem 7. It can, in fact, be shown that the two tests are equivalent for large samples.
EXAMPLE 21

Mendelian theory indicates that the shape and color of a certain variety of pea ought to be grouped into four groups, "round and yellow," "round and green," .. angular and yellow;' and angular and
U

446

TESTS OF HYPOTHESES

IX

green," according to the ratios 9/3/3/1. For n = 556 peas, the following were observed (the last column gives the expected number): Round and yellow Round and green Angular and yellow Angular and green 315 108 101 32 312.75 104.25 104.25 34.75
9 1 6'

A size-.05 test of the null hypothesis .Yf0: PI = and P4 = /6 is given by the following: Reject.Yf if and only if Q~ 7.81. The observed Q~ is
=

P2

3 1 6'

P3

3 1 6'

L
1

(N. _ npO)2
J

npj

exceeds XT-alk)

X~95(3)

(315 - 312.75)2 (l08 - 104.25)2 (101 - 104.25)2 (32 - 34.75)2 312.75 + 104.25 + 104.25 + 34.75
~

.470,

and so there is good agreement with the null hypothesis; that is, there is a good fit of the data to the model. IIII Theorem 8 can be generalized to the case where the probabilities Pj may depend on unknown parameters. The generalization is given in the next theorem. Theorem 9 Let the possible outcomes of a certain random experiment be decomposed into k + 1 mutually exclusive sets, say AI' ... , A k + 1 Define Pj = P[Ajl, j = 1, ... , k + 1, and assume that Pj depends on r unknown parameters 01 , , On SO that Pj = /tiOl' ... , Or), j = 1, ... , k + 1. In n independent repetitions of the random experiment, let N j denote the number of outcomes belonging to set A j ' j = 1, ... , k + 1, so
k+l

that

Nj

= n. Let 9 1 ,

j= 1

9 r be BAN estimators (e.g., maximum ,

likelihood estimators) of 01 , , Or based on N 1 , certain general regularity conditions on the /t / s,

Nk

Then, under

' _ k~l (N j - nPj )2 Qk - L - . . . .

j=l

nPj

(28)

has a limiting distribution that is the chi-square distribution with k - r degrees of freedom, where Pj = /ti91' ... , 9 r ), j = 1, ... , k + 1. IIII

CHI-SQUARE TESTS

447

The proof of Theorem 9 is beyond the scope of this book. The limiting distribution given in Theorem 9 differs from the limiting distribution given in Theorem 8 only in the number of degrees of freedom. In Theorem 8 there are k degrees of freedom, and in Theorem 9 there are k - r degrees of freedom; the number of degrees of freedom has been reduced by one for each parameter that is estimated from the data. No mention of hypothesis testing is made in the statement of Theorem 9. However, we will show now how the results of the theorem can be used to obtain a goodness-of~fit test. Suppose that it is desired to test that a random sample Xt, ... , Xn came from a density I(x; 81, ... , 8,.), where 81, ... , 8,. are unknown parameters but the function I is known. The null hypothesis is the composite hypothesis .Yf0: Xi has density I(x; 81, ... , 8,.) for some 81, ... , 8,.. The null hypothesis states that the random sample came from the parametric family of densities that is specified by I( . ; 81 , , 8,.). If the range of the random variable Xi is decomposed into k + 1 subsets, say AI' ... , A k + l , if Pj = P[X, E A j ], and if N j = number of X/s falling in A j' then, according to Theorem 9,

Q" = L 1

k+l

(N - np.)2
j

nPj

is approximately distributed as the chi-square distribution with k - r degrees of freedom if n is large and .Yf 0 is true, where Pj = /tiel, ... , e,.) and i is a maximum-likelihood estimator of 8i , i = 1, ... , r, obtained from the statistics N l , , N k {Note that /ti81' ... ,8,.) =P[XiEA j ], which for a continuous random variable Xi equals SA.jl(x; 81 , , 8,.) dx.} Hence, a test of .Yf0 can be obtained by rejecting .Yf0 if and only if the statistic Qfc is large; that is, reject .Yf0 if and only if exceeds xi -a(k - r), where xi -a(k - r) is the (1 - ct)th quantile of the chi-square distribution with k - r degrees of freedom. Such a test is called a goodness-of-fit test since it tests whether or not the observations Xl, . , Xn fit, or are consistent with, the assumption that they are observations from the density I(x; 81 , , 8..). In the above, the 8" for i = 1, ... , r, were estimated by using the statistics N l , . , Nk rather than Xl' , X n The statistics N l , . , . , Nk give the number of X observations falling in each of the A j subsets or groups. In practice, often the values of the X/s are not recorded, and then the group totals N l , . , Nk constitute the available information. If, however, the observations Xl' ... , Xn were available, then one could estimate 8i , i= 1, ... , r, more efficiently by using, say, maximum-likelihood estimators based on Xl ... , X n When such estimators are used, the limiting distribution of Q~ is no longer a chi-square distribu~ tion with k - r degrees of freedom; instead, the limiting distribution of Qi is bounded between a chi-square distribution with k - r degrees' of freedom and a

Q"

448

TESTS OF HYPOTHESES

IX

chi-square distribution with k degrees of freedom. In a sense, some of the "lost" r degrees of freedom are recouped by efficiently estimating (h, ... , Or. For a proof of Theorem 9 and further discussion of the above, the reader is referred to Kendall and Stuart [14]. EXAMPLE 22 Suppose it is desired to test the hypothesis that an observed random sample XI, , Xn has been drawn from some normal population. Let the n sample values Xl' . , Xn be grouped into k + 1 classes. For example, thejth class could be taken as all those observations falling in the interval (Zj_I' Zj], j = 1, ... , k + 1, for some Zo < Zl < Z2 < ... <Zk < zk+l,where Zo = - 00 and Zk+1 = + 00. Then

Let {l and ff be the maximum-likelihood estimates of Jl and (1 based on nl, ... , nk, where nj is the number of observations falling in thejth interval. Then,

can be determined from the sample, and so the value

qk = I

k+ I
j= I

(n - nftJ)2
J

nPJ

"

of Q~ can also be obtained from the sample. The hypothesis that the sample came from a normal population would be rejected at the ex lever if q/c > xI -rlk - 2). If, on the other hand, Jl and (1 were obtained from maximum..likelihood estimators based on X., ... , X n , then the asymptotic distribution of Q~ would fall between a chi-square distribution with k - 2 degrees of freedom and a chi.. square distribution with kdegrees of freedom. The hypothesis would be rejected if q~ > C, where C falls between xi _rlk - 2) and XI -rJ.(k). Note that for k large there is little difference IIII between XI-rlk - 2) and xf-rlk).

5.3

Test of the Equality of Two Multinomial Distributions and Generalizations

A problem that is of great practical importance is that of testing whether several random samples can be considered as drawn from the same population. For instance, in Subsec. 4.3 we tested whether several assumed normal populations could be considered the same normal population. In this subsection we first

CHI-SQUARE TESTS

449

indicate a test of the hypothesis that two multinomial populations can be considered the same and then indicate some generalizations. Suppose that there are k + 1 groups associated with each of the two multinomial populations. Let the first popula ti on have associated probabiIi ties Pll' P12' ... , PI k' PI, k+ 1 and the second P21' P22' ... , P2k , P2, k+l It is desired to test .Yt0: P1l = P2j (= Pj' say), j = 1, ... , k + 1. For a sample of size nl from the first population, let N 1j denote the number of outcomes in group j, j = 1, ... , k + 1. Similarly, let N 2j denote the number of outcomes in group j of a sample of size n2 from the second population. (Here we are assuming that the sample sizes nl and n2 are known.) We know that
k+l (N ij - niPij )2 j= 1 nj Pij

has a limiting chi-square distribution with k degrees of freedom for i = 1 and 2; hence

I2 k+ I

(N

i=1 j=1

ij - niPij niPij

)2

has a limiting chi-square distribution with 2k degrees of freedom if the two random samples are independent. If.Yt 0 is true, then
Q2A;

i=1 j=1

I2 k+l(N I ij -

n p)2 i j niPj

(29)

has a limiting chi-square distribution with 2k degrees of freedom. If.Yt 0 specifies the values Pj' then Q2k is a statistic and can be used as a test statistic. On die other hand, if the Pj defined by .Yt0 are unknown, then they have to be estimated. If.Yt 0 is true, the two samples can be considered as one random sample of size n 1 + n 2 from a multinomial population with probabilities PI, ... , Pk+l. Maximum-likelihood estimators of the Pj are then (N 1j + N 2j )/(nl + n2)' j = 1, ... , k, and if the Pj in Eq. (29) are replaced by their maximum-likelihood estimators, we then obtain
Q;k = kII [Nij - ni(N Ij + N 2j)/(nl + n2)F . i= 1 j= 1 ni(N 1l + N 2j)/(n 1 + n2)

(30)

It can be shown that Q;k has a limiting chi-square distribution with 2k - k = k degrees of freedom. (This result is not a direct corollary of Theorem 9; it would, however, be a corollary of a generalization of Theorem 9 from one to two populations.) Again the degrees of freedom of the limiting distribution of Q2k have been reduced by unity for each parameter estimated.

450

TESTS OF HYPOTHESES

IX

Another test of the homogeneity of two multinomial populations can be derived by finding the generalized likelihood-ratio A and employing Theorem 7 to obtain the limiting distribution of - 210g A. (Reparameterization is required before Theorem 7 can be employed directly.) The details of finding such a test are left as an exercise. EXAMPLE 23 In an opinion survey regarding a certain political issue there was some question as to whether or not the eligible voters under 25 years of age might view the issue differently from those over25. Fifteenhundred individuals of those over 25 were interviewed, and 1000 of those under 25 were interviewed with the following results (the data are obviously artificial to facilitate calculations):
Opposed Under 25 Over 25 400 600 1000 Undecided 100 400 500 Favor 500 500 1000 Total 1000 1500 2500

Total

Test the null hypothesis that there is no evidence of difference of opinion due to the different age grouping; that is, test .Yf0: PI} = P2j = p} , j = 1, 2, 3. PI and P2 need to be estimated. We can calculate the value of the statistic given in Eq. (30) as follows: (400 - 1000 . 1000/2500)2 (100 - 1000 500/2500)2 + 1000 . 500/2500 1000 1000/2500 (500 - 1000 . 1000/2500)2 (600 - 1500 . 1000/2500)2 + 1000 1000/2500 + 15001000/2500 (400 - 1500 . 500/2500)2 (500 - 1500 10oo/2500? + 1500 500/2500 + 1500' 1000/2500

= 125 .

The 99 percent quantile point for the chi-square distribution with two degrees of freedom is only 9.21; so there is strong evidence thatthetwoage groups have different opinions on the political issue. fill The technique presented in this subsection can be generalized in two directions. First, a test of the homogeneity of several, rather than just two, multinomial populations can be obtained, and, second, a test of the hypothesis that

CHI-SQUARE TESTS

451

several given samples are drawn from the same population of a specified type (such as the Poisson, the gamma, etc.) can be obtained using a procedure similar to that above. We illustrate with an example.

EXAMPLE 24 One hundred observations were drawn from each of two Poisson populations with the following results:
0 Population 1 Population 2 Total
11

2 28 28 56

3 20 17 37

4 9 11 20

8 1 0

9 or more 0 0

Total 100 100 200

25 27 52

3 1

3 2
11

0 1

13 24

Is there strong evidence in the data to support the contention that the two Poisson populations are different? That is, test the hypothesis that the two populations are the same. This hypothesis can be tested in a variety of ways. We first use the chi-square technique mentioned above. We group the data into six groups, the last including all digits greater than 4, as indicated in the above table. If the two populations are the same, we have to estimate one parameter, namely, the mean of the common Poisson distribution. The maximum-likelihood estimate is the sample mean, which is 0(24)

+ 1(52) + 2(56) + 3(37) + 4(20) + 5(4) + 6(5) + 7(1) + 8(1)

200

420 = 200 = 2.1. The expected number in each group of each population is given by
0 12.25
1

2 27.00

3 18.90

4 9.92

5 or more 6.21

25.72

The value of the statistic in Eq. (29), where niPj is replaced by the estimates given in the above table, can be calculated. It is approximately 1.68. The degrees of freedom should be 2k - 1 (one parameter is estimated), which is 9. The test indicates that there is no reason to suspect that the two assumed Poisson populations are different Poisson populations. IIII

452

TESTS OF HYPOTHESES

IX

We mentioned earlier that there are several methods of testing the null hypothesis considered here. For example the generalized likelihood-ratio principle and employment of Theorem 7 yield a test that the student may find instructive to find for himself.

5.4

Tests of Independence in Contingency Tables

A contingency table is a multiple classification; for example, in a public opinion survey the individuals interviewed may be classified according to their attitude on a political proposal and according to sex to obtain a table of the form
Favor Men Women
1154 1083

Oppose
475 442

Undecided
243 362

This is a 2 x 3 contingency table. The individuals are classified by two criteria, one having two categories and the other three categories. The six distinct classifications are called cells. A three-way contingency table would have been obtained had the individuals been further classified according to a third criterion, say, according to an annual-income group. If there were five income groups set up (such as under $2000, $2000 to $4000, ... ), the contingency table would be called a 2 x 3 x 5 table and would have 30 cells into which a person might be put. It is often quite convenient to think of the cells as cubes in a block two units wide, three units long, and five units deep. If the individuals were still further classified into eight geographic locations, one would have a four-way (2 x 3 x 5 x 8) contingency table with 240 cells in a four-dimensional block with edges two, three, five, and eight units long. A contingency table provides a convenient display of the data for ultimately investigating suspected relationships. Thus one may suspect that men and women will react differently to a certain political proposal, in which case one would construct such a table as the one above and test the null hypothesis that their attitudes were independent of their sex. To consider another example, a geneticist may suspect that susceptibilitytoacertain disease is heritable. He would classify a sample of individuals according to (i) whether 01 not they ever had the disease, (ii) whether or not their fathers had the disease, and (iii) whether or not their mothers had the disease. In the resulting 2 x 2 x 2 contingency table he would test the null hypothesis that classification (i) was independent of (ii) and (iii). Again a medical research worker might suspect a certain environmental condition favored a given disease and classify individuals according to (i) whether or not they ever had the disease,

CHI-SQUARE TESfS

453

An industrial engineer could use a contingency table to discover whether or not two kinds of defects in a manufactured product were due to the same underlying cause or to different causes. It is apparent that the technique can be a very useful tool in any field of research.

(ii) whether or not they were subject to the condition.

Two-way contingency tables We shall suppose that n individuals or items are classified according to two criteria A and B, that there are r classifications AI, A 2 , . , A,. in A and s classifications B 1 , B 2 , Bs in B, and that the number of individuals belonging to Ai and B} is Nij. We have then an r x scontingency table with cell frequencies N ij and I N i} = n:
0 ,

(31)

As a further notation we shall denote the row totals by N i. and the column totals by N .}; that is,
NI.

=" .t.- N
}

'J

and

N oj

=" .t.- N .. ,
I}

Of course,

It

N i.

= 2: N. j = n.
J

We shall now set up a probability model for the problem with which we wish to deal. The n individuals will be regarded as a sample of size n from a multinomial population with probabilities Pi} (i = 1, 2, ... , r; i = 1, 2, .. " s). The probability density function for a single observation is

where or
1

and

L xi} = L t, }

454

TESTS OF HYPOTHESES

IX

We wish to test the null hypothesis that the A and B classifications are independ~ ent, i.e., that the probability that an individual falls in B j is not affected by the A class to which the individual happens to belong. Using the symbolism of Chap. I, we would write and or
P[A i n B}l
=

P[AilP[Bjl.

If we denote the marginal probabilities P[Ail by Pi. (i = 1, 2, ... , r) and the marginal probabilities P[Bjl by P.j (j = I, 2, ... , s), the null hypothesis is simply

Pi.

= 1,

P.j

= 1.

(33)

When the null hypothesis is not true, there is said to be interaction between the two criteria of classification. The complete parameter space 8 for the distribution of N l l , . , N rs has rs - 1 dimensions (having specified all but one of the Pi)' the remaining one is fixed by I Pi} = 1), while under :Yf 0 we have a parameter space 8 0 with i,j r - 1 + s - 1 dimensions. (The null hypothesis is specified by Pi., i = I, ... , r, and P.), j = 1, ... , s, but there are only r - 1 + s - I dimensions because I Pi. = I and I P.j = 1.) The likelihood for a sample of size n is

L=np?y
i, }

and its maximum in (3 occurs when

In 8

0 ,

L =

n(Pi. p.)nu = (0 P?:) (n P~/),

I,) I }

(34)

and its maximum occurs at and The generalized likelihood-ratio is therefore

P.} = - .
n

n.}

(35)

(36)

em-SQUARE TESTS

455

The distribution of A under the null hypothesis is not unique because the hypothesis is composite and the exact distribution of A does involve the unknown parameters Pi. and P.j; hence, it is very difficult to solve for 20 in sup P 8[A ~ 20] =

eo

ex. For large samples we do have a test, however, because - 2 log A is in that case approximately distributed as a chi-square random variable with
rs - 1 - (r

+s-

2)

= (r -

1)(s - 1)

degrees of freedom and on the basis of this distribution a unique critical region for 1 may be determined. The degrees of freedom rs - 1 - (r + s - 2) is obtained by subtracting r + s - 2, which is the dimension of 9 0 , from ra - 1, which is the dimension of 9. Also, (r - 1)(a - 1) is the number of parameters specified by :Yt0 (See Theorem 7 and the comment following it.) Actually, the null hypothesis :Yto: Pij = Pi.P.j is not of the form required by Theorem 7; so it might be instructive to consider the necessary reparameterization. For convenience, let us take r = s = 2. Now (3 = {(el , O 2 , ( 3 ) = (PH' Po, P21): Pu > 0; PI2 > 0; P21 ~ 0; and P11 + PI2 + P21 :::;;; I}. Let 8 with points ) denote the reparameterized space, where O~ = PI I - Pl. P.I' 0; = PI., (e~, 0; , ( 3 and OJ = P.I' It can be easily demonstrated that 9' is a one.. to.. one transformation of 9. Also, the null hypothesis :Yt0: Pu = P1. P.I, Pu = P1.(1 - P.I), and P21 = (1 - PI )P.I in the original parameter space 9 becomes :Yt~: O~ = 0 and 0; and 0; unspecified in the reparameterized space 9'. [Note that PI2 = P1.(l - P.I) is equivalent to PI. - PH = PI.(1 - P.I)' which is equivalent to P11 P1.P.I = O. Similarly for PZI = (1 - P1.)P.I] :Yt~ is of the form required by Theorem 7. In general, a point in the (ra - l)-dimensional parameter space 9 can be conveniently displayed as
1

PH P21

PI2 PZZ P,.-I,2 Pr2

PI,s-1 P2, s-1 Pr-I.s-l Pr,s-l

Ph PZs
Pr-l, s

P,.-I.I Prl

and a point in the reparameterized space 9' can be displayed as

PH - P1.P.l P21 - PZ.P.l P,.-I, 1
-

P12 - P1.P.2 P22 - PZ.P.2 P,.- I, 2

-

Pl,8-1 - P1.P., s-1 Pz, s-1 - P2.P., s-1 P,.-I,8-1 - P,.-I,.P.,s-1 P.,s-1

PI. P2. P,.-I .

P,.-I,.P.l P.l

p,.- 1 ,.P.2

P.2

456

TESTS OF HYPOTHESES

IX

In casting about for a test which may be used when the sample is not large, we may inquire how it is that a test criterion comes to have a unique distribution for large samples when the distribution actually depends on unknown parameters which may have any values in certain ranges. The answer is that the parameters are not really unknown; they can be estimated, and their estimates approach their true values as the sample size increases. In the limit as n becomes infinite, the parameters are known exactly, and it is at that point that the distribution of A actually becomes unique. It is unique because a particular point in 8 0 is selected as the true parameter point, so that the N ij are given a unique distribution, and the distribution of A is then determined by this distribution. It would appear reasonable to employ a similar procedure to set up a test for small samples, i.e., to define a distribution for A by using the estimates for the unknown parameters. In the present problem, since the estimates of the Pi. and P.j are given by Eq. (35), we might just substitute those values in the distribution function of the N lj and use the distribution to obtain a distribution for A. However, we should still be in trouble; the critical region would depend on the marginal totals Nt. and N. j ; hence the probability of a Type I error would vary from sample to sample for any fixed critical region 0 < A < Ao. There is a way out of this difficulty, which is well worth investigation because of its own interest and because the problem is important in applied statistics. Let us denote the joint density of all the N ij briefly by f(nt), the marginal density of all the Nt. and N. j by g(ni" n.}), and the conditional density of the N i}, given the marginal totals, by
f( ni}ni.,n.j )= I

( ) g nt., n.}

!(ni})

Under the null hypothesis, this conditional distribution happens to be independent of the unknown parameters (as we shall show presently); the estimators N i./n and N ,}In form a sufficient set of statistics for the Pt, and P.j' This fact will enable us to construct a test. The joint density of the N ij is simply the multinomial distribution
fen/i) =f(nl1, n12, ... , n,,) =

It
i. J

nij

! n]l;'Y
t, }

(37)

in e, and in 8 0 (we are interested in the distribution of A under :if0) this becomes
f(n", nt., .. , n,.) = nn! ,
nij'

(0 P~.i') (I1 p~l)'

~

(38)

t, j

To obtain the desired conditional distribution, we must first find the distribution

em-SQUARE TESTS

457

of the N i. and N.}, and this is accomplished by summing Eq. (38) over all sets of nil such that (39) and "n,.=n .. " ~} I. ~ n I} = n}
i
j

For fixed marginal totals, only the factor l/flnjj! in Eq. (38) is involved in the sum; so we have, in effect, to sum that factor over all nij subject to Eq. (39). The desired sum is given by comparing the coefficients of fl x';'. in the expression
i

(Xl

+ ... + xrfoJ(xl + ... + x r)n.

On the right-hand

+ ... + xrt = (Xl + ... + Xr)n. side the coefficient of fl Xii. is simply
2

(Xl

(40)

n! fl ni.!
i

(41)

On the left-hand side there are terms with coefficients of the form
n. l ! n.2J

n'
s

=}=--_

fl . n.j!

fl nil! fl ni2! fl nisI i, fl nij! ' i i i j

(42)

where nij is the exponent of Xi in the jth multinomial. In this expression the nij satisfy conditions of Eq. (39); the first condition is satisfied in view of the multinomial theorem, while the second is satisfied because we require the exponent of Xi in these terms to be ni.. The sum of all such ,coefficients, Eq. (42), must equal Eq. (41); hence, we may write
n!

(\I nl.! I} n.,! ) .

(43)

This is precisely the sum that we require because there is obviously one and only one coefficient of the form of Eq. (42) on the left of Eq. (40) for every possible contingency table, Eq. (31), with given marginal totals. The distribution of the N i. and N ,} is, therefore,
g( n
i.,

n) .} -

t)2 (fl Pi. ni')(f] n. /) (fl ni.(n !)(f] n.}!) P.j'

(44)

which shows, incidentally, that the N i are distributed independently of the N. j under :Yf 0; this is unexpected because N 1. and N. I , for example, have the random variable Nll in common! The conditional distribution of the N ij' given the marginal totals, is obtained by dividing Eq. (38) by Eq. (44) to obtain (flni. !)(fln.)!) f(n ll , n12 , ... , nrs I nl.' n2., .. " n.s) = 'fl " (45) n. nij.

458

TESTS OF HYPOTHESES

IX

lr-------------~-----------

which, happily, does not involve the unknown parameters and shows that the estimators are sufficient. To see how a test may be constructed, let us consider the general situation in which a test statistic A for some test has a distribution/A(A; 8) which involves an unknown parameter 8. If 8 has a sufficient statistic, say T, then the joint density of A and T may be written
fA,T(A, t; 8) =/AIT(Alt)/T(t; 8),

and the conditional density of A, given T, will not involve 8. Using the conditional distribution, we may find a number, say Ao(t), for every t such that

A.o(t)

/AIT(AI t) dA

= .05,

(46)

for example. In the At-plane the curve A = Ao(t) together with the line A = 0 will determine a region R. See Fig. 7. The probability that a sample will give rise to a pair of values (A, t) which correspond to a point in R is exactly .05 because
P[(A, T)
E

R]

J J fA, T(A, t; 8) dA dt -co = roo [('ImO. It) d.l.]fT(t;

A.o(t)

co

0) d t

Jco .05fT(t; 8) dt - co

=.05. Hence we may test the hypothesis by using Tin conjunction with A. The critical region is a plane region instead of an interval 0 < A < Ao; it is such a region that, whatever the unknown value of 8 may be, the Type I error has a

em-SQUARE TESTS

459

specified probability. The test in any given situation actually amounts to a conditional test; we observe T and then perform the test by using the interval o < A < Ao(t) using the conditional distribution of A, given T. !tis to be observed that this device cannot be employed unless there is a sufficient statistic for e. The above technique is obviously applicable when e is a set of parameters rather than a single parameter and has a set of sufficient statistics. In particular, the technique may be employed to test the null hypothesis of a two-way contingency table using Eq. (36) to define A. One merely uses the conditional distribution of Eq. (45) and determines an interval 0 < A < Ao(ni.; n.j) which has the desired probability of a Type I error for the observed marginal totals. In applications of this test one is confronted with a very tedious computation in determining the distribution of A unless r, s, and the marginal totals are quite small. It can be shown, however, that the large-sample approximation may be used without appreciable error except when both rand s equal 2. In the latter instance, other simplifying approximations have been developed (see, for example, Fisher and Yates, "Tables for Statisticians and Biometricians," Oliver & Boyd Ltd., Edinburgh or London, 1938), but we shall not explore the problem that far. Another test of the :Yf 0 given in Eq. (33) is obtained if the distribution in Eq. (45) is replaced by its multivariate normal approximation since then it can be shown that the statistic
Q

= I [Nij - n(Ni./n)(N.j!n)F
i,j

n(Ni./n) (N.j!n)

(47)

has approximately the chi-square distribution with rs - 1 - (r - 1 + s - 1) = (r - 1)(s - 1) degrees offreedom. The test criterion is to reject :Yf 0 for large Q. This is the criterion first proposed (by Karl Pearson) for testing the hypothesis, and it differs from -2 log A by terms of order I/Jn. The two criteria are therefore essentially equivalent unless n is small. The argument that Q is a reasonable test statistic is entirely analogous to that used in Subsec. 5.2 above to justify Eq. (25). The statistic Q of Eq. (47) has intuitive appeal. Nij is the observed numberin the ijthcell, and n(N i.!n)(N.j/n) isan estimator of the expected number in the ijth cell when :Yf 0 is true. Thus, Q will tend to be small for :Yf 0 true and large for :Yf 0 false.

If the elements of a population can be classified according to three criteria A, B, and C with classifications Ai (i = 1, 2, ... , S1)' B j (j = 1, 2, ... , S2), and Ck (k = 1, 2, ... , S3), a sample of n individuals may be classified in a three-way S1 x S2 X S3 contingency table. We shall let Pijk

Three-way contingency tables

460

TESTS OF HYPOTHESES

IX

represent the probabilities associated with the individual cells and Ntlk be the numbers of sample elements in the individual cells, and, as before, marginal totals will be indicated by replacing the summed index by a dot; thus and (48)

There are four hypotheses that may be tested in connection with this table. We may test whether all three criteria are mutually independent, in which case the n utI hypothesis is
Pilk

= PI .. P.l.P ..k'
i

(49)

where

Pi .. =

L L Pilk'
j

P.j. =

L L Pijk' i
k

and

P ..k

= L L Pijk;
1

or we may test

whether anyone of the three criteria is independent of the other two. Thus to test whether the B classification is independent of A and C, we set up the null hypothesis
Pijk = Pl.k P.}.,

(50)

where

Pi.k

L
}

Pijk'

The procedure for testing these hypotheses is entirely analogous to that for the two-way tables. The likelihood of the sample is

L=
where
i. j. k

i,j. k

II p7j,\
nijk = n.

(51)

PUk =

and

i.}. k

In

(3

the maximum of L occurs when

so that

To test the null hypothesis in Eq. (50), for example, we make the substitution of Eq. (50) into Eq. (51) and maximize L with respect to the PUr. and P.). to find
" nt.k Plk=. n

and

Pj ..

"

n .j. =-, n

and sup L =
i!o

~n (II n7:ir. n
I, If.

k )

(II n~l')'
j

(53)

TESTS OF HYPOTHESES AND CONFIDENCE INTERV AIS

461

The generalized likelihood-ratio). is given by the quotient of Eqs. (52) and (53), and in large samples - 2 log A has the chi-square distribution with
8 18 2 83 -

1-

[(8183 -

1)

+ 82 -

1]

= (8183 - 1)(82 -

1)

degrees of freedom. Again the large-sample distribution is quite adequate for many purposes. (8183 -1) +(82 -1)isthedimensionofB o , and 8 18 2 83 -1 is the dimension of t). A test statistic analogous to that given in Eq. (47) for testing independence in a 2 x 2 contingency table can also be derived. F or testing :If 0: A and C classifications are independent of the B classification, such a test statistic is

Q=

L L L [N ijk i
j

n(N u /n)(N. j ./n)]2 . n(Ni.k/n)(N.J./n )

(54)

Under :lfo , Q has an asymptotic chi-square distribution with 818283 -1 - (8183 - 1) - (82 - 1) = (8183 - 1)(82 - 1) degrees of freedom. Again, the statistic Q of Eq. (54) has intuitive appeal since N ijk is the observed number in cell ijk and n(Ni.kln)(N.j.!n) is an estimator of the expected number when :If 0 is true.

TESTS OF HYPOTHESES AND CONFIDENCE INTERVALS

In Subsec. 3.4 above we noted that a confidence interval for a unidimensional parameter e could be used to obtain a test of :If 0: e = eo versus :If1: e =1= eo. In this section we will further explore that concept and show that one can reverse the operation; that is, one can use a family of tests of :If 0: e = eo versus :lf1 : e =1= eo (the family is generated by varying eo) to obtain a confidence interval for e. Our considerations in this section will not be very thorough; our intent is merely to present an introduction to the usefulness of the close relationship between hypothesis testing and confidence intervals. Our discussion can be made somewhat more general if we speak in terms of confidence 8et8 rather than confidence interval8. As usual, let X denote the sample space, e the parameter space, and (Xh ... , x,,) the observed sample.

Definition 18 Confidence set A family of subsets of the parameter space t) indexed by (Xl' ... , x,,) E X, denoted by 9 = {t)(Xl' ... , x,,): S(Xl' ... , x,,) c e; (Xl"'" X,,) E X}, is defined to be afamity of confidence sets with confidence coefficient y if and only if
P9[9(Xh ... , X,,) contains e]

for all

S.

(55)

1//1

462

TESTS OF HYPOTHESES

IX

It should be emphasized that any member, say 8(Xl' ... , X,.), of the family of confidence sets is a subset of 8, the parameter space. 8(X1 , , Xn) is a random subset; for any possible value, say (Xl' ... , X,.), of (Xl' ... , X,.), 8(Xl' ... , X,.) takes on the value 8(Xl' ... , X,.), a member of the family 9. To aid in the interpretation of the probability statement in Eq. (55), note that for a fixed (yet arbitrary) fJ "8 (Xl' ... , X,.) contains fJ" is an event [it is the event that the random interval 8(Xl' ... , X,.) contains the fixed fJ] and the fJ that appears as a subscript in P8 is the fJ that indexes the distribution of the X/s appearing in 8(Xl' ... , X,.). For instance, suppose Xl' ... , X,. is a random sample from N(fJ, 1). 8 = {fJ; - 00 < fJ < oo}. Let the subset 8(x 1 , , X,.) be the interval

(x - z/Jn, x + z/J~), where z is given by cI>(z) - cI>( -z) = y; then the family of subsets 9 = {8(Xl' ... , xJ: 8(Xh ... , X,.) = (x - z/J~, x + z/Jn)} is a family
of confidence sets with a confidence coefficient y since

P.[9(X" ... , X.) contains 0] = P.[x -

J~ < 0 < X + ~]
=

-fJ] = P8 [ -z < X l/Jn < z

for all fJ

8.

The family 9 is a family of confidence intervals for fJ having a confidence coefficient y. In general, then, a confidence interval is an example of a . confidence set. Confidence sets can be constructed from tests of hypotheses, as we nOW show. Let 180 be a size-ct test (nonrandomized) of the null hypothesis .1f0: fJ = fJo, and let X(fJ o) be the acceptance region of the test 180 [The acceptance region is the set complement of the critical region; that is, if the critical region is given by C(fJ o) , then X(fJ o) = X - C(fJ o)'] Note that X(fJ o) is a subset of X indexed by fJ o Since the test 180 has size ct,
P80 [(X1 ,
,

X,.)

X(fJ o)] = 1 - ct.

If we nOw vary fJo over B and for each fJo we have a test 180' then we get a family of acceptance regions, namely, {X(fJ o): fJo E 8}. X(fJ o) is the acceptance region of test 180' One can now define
8(Xb ... , Xli)

= {fJo: (Xl' ... , X,.) E X(fJ o)}.

(56)

Clearly 8(x1 , , X,.) is a subset of 8. Furthermore, the family {B(Xl' ... , X,.)} is a family of confidence sets with a confidence coefficient y = 1 - ct since One has {8(Xl' ... , XJ contains fJo} if and only if one has {(Xb ... , X,.) E X(fJ o)}, and so
P80[8(Xl' ... , X,.) contains fJo]

= P80 [(X1 ,

Xn)

X(fJ o)]

= 1-

ct.

TESTS OF HYPOTHESES AND CONFIDENCE INTERVALS

463

EXAMPLE 25 Let Xl' ... , X" be a random sample from N(fJ, 1), and consider testing .1f0: fJ = fJ 0 A test with size rl is given by the following: Reject
.1f0 if and only if Ix - fJo I z/j~, where z is defined by 4l(z) - 4l( -z) = 1 - rl. The acceptance region of this test is given by

30(110) = {(x" ... , x.); 110 We can now define, as in Eq. (56),
S(XH ... , xn)

,in < x< 110 + ,in}.

, xn)
E

= {fJ o: (XH

X(fJo)}

= {1I0: 110 =

,in <i < 110 + ,in} {1I0: x - j;; < 110 < x + ,in}.
////

S(Xl' ... , xJ is a confidence set (in fact a confidence interval) with a

confidence coefficient y = 1 -

rl.

The general procedure exhibited above shows how tests of hypotheses can be used to generate or construct confidence sets. The procedure is reversible; that is, a given family of confidence sets can be "reverted" to give a test of hypothesis. Specifically, for a given family {e(Xl' ... , xn)} of confidence sets with a confidence coefficient y, if we defined

then the nonrandomized test with acceptance region X(fJ o) is a test of .1f0: fJ = fJo with size rl = 1 - y. The usefulness of the strong relationship between tests of hypotheses and confidence sets is exemplified not only in the fact that One can be used to construct the other but also in the result that often an optimal property of one carries over to the other. That is, if one can find a test that is optimal in some sense, then the corresponding constructed confidence set is also optimal in some sense, and conversely. We will not study the very interesting theoretical result alluded to in the previous sentence, but we will give the following in order to give some idea of the types of optimality that can be expected. (See the more advanced books of Ref. 16 and Ref. 19 for a detailed discussion.) An optimum property of confidence sets is given in the following definition.

464

TESTS OF HYPOTHESES

IX

Definition 19 Uniformly most accurate A family {9*(x 1, ... , xn)} of confidence sets with a confidence coefficient y is defined to be a uniformly most accurate family of confidence sets at a confidence coefficient y if for any other family {9(X1' .,., xn)} of confidence sets with a coefficient y
P8[9*(X1, ... , Xn) contains 0'] < P8[9(X1, ... , Xn) contains 0']

for all 0 and 0'.

IIII

Definition 19 is saying that 9*(Xb .. , Xn) is less likely to contain an incorrect 0' than is 9(X1, .. , X n), whereas both 9*(X1, . , Xn) and 8(X1, . .. , Xn) have the same probability of containing the correct O. As you may have guessed, uniformly most accurate confidence sets rarely exist. However, uniformly most accurate confidence sets within restricted classes of confidence sets could also be defined, and then one could be hopeful of the existence of such optimal confidence sets. A general type of result that derives from the close relationship between tests of hypotheses and confidence sets is the following: If y* is a uniformly most powerful size-a test of :Yf 0: 0 = 00 within some restricted class of tests, then the confidence set corresponding to y* is uniformly most accurate with coefficient y = 1 - a within some restricted class of confidence sets. With such a result one can see how an optimality of a test can be transferred to an optimality of a corresponding confidence set, and therein lies the real utility of the close relationship between hypotheses testing and confidence sets.

7
7.1

SEQUENTIAL TESTS OF HYPOTHESES

Introduction

Sequential analysis refers to techniques for testing hypotheses or estimating parameters when the sample size is not fixed in advance but is determined during the course of the experiment by criteria which depend on the observations as they occur. In this section we propose to consider, and then only briefly, one form of sequential analysis, namely, the sequential probability ratio test. In Sec. 2 above we considered testing the simple null hypothesis :Yf 0: 0 = 00 versus the simple alternative hypothesis :Yf1: 0 = 01 , It was shown (NeymanPearson lemma) that for samples of fixed size n, the test which minimized the size, say p, of the Type II error for fixed size, say a, of the Type I error was a simple likelihood-ratio test. That is, for fixed n and a, p was minimized. Suppose now that it is desired to fix both a and P in advance and then find that simple likelihood-ratio test having minimum sample size n and having size of Type I error equal to a and size of Type II error p. The solution of such a problem is illustrated in the following example.

SEQUENTIAL TESTS OF HYPOTHESES

46S

EXAMPLE 26 A manufacturer of a certain component, say, an oil seal, knows the history of his current manufacturing process. He knows, for instance, that the distribution of lifetimes of the seals now being manufactured is, say, N(lOO, 100). A new manufacturing process is suggested; the manufacturer wants to continue with his present manufacturing process if the new process is not better (longer mean lifetime), yet he also wants to be quite certain to switch to the new process if the new process increases the mean lifetime by, say,S percent. He proposes to take a sample of observations of lifetimes of seals made by the new process and then from the sample decide whether or not the process has longer mean life. He models the experiment by assuming that the random variable X, representing the lifetime of a seal manufactured using the new process, is distributed as NCO, 100), and he wants to test .1P0: 0 ::::; 100 versus .1P1: 0 > 100. He fixes his error sizes and wants to determine the sample size n so that, say,
.01 = ex

= P8=100[reject .1P0]

and

.05

= fJ = P 9 =10S[accept :Yf 0]'

That is, he seeks to determine n so that there is only a 1 percent chance of rejecting that the new process is no better than the old when it is not, yet there is a 95 percent chance of rejecting that the mean lifetime of the new process is less than 100 when in fact it is 5 percent larger. It can be shown that the simple likelihood-ratio test is equivalent to the test of rejecting :Yf 0 for large X n Thus he seeks to determine n and k so that and or
.01
=

1_

<f>(k -

100) IO/y' n

and

.05

= <f> (

k - 105) ! / IO/yn

<f>(k implies

IOIJn
k -100
10/yn
and
<f> (

100) = .99
~

r:

2.326,

k - 105) 1)1, = .05 10

466

TESTS OF HYPOTHESES

IX

implies

k - 105 ;:
lO/y n

-1.645;

which together imply that 100 + 10(2.326)IJ~ ~ 105 - 10(1.645)IJn) or n ~ 63.08; so a sample of size 64 is needed. II I I Referring to the above example, the following considerations make sequential analysis interesting both from the theoretical and practical viewpoint. In drawing the 64 observations to test .Yf0, it is possible that among the first few observations, say 20, 30, or 40, the evidence is quite sufficient relative to ct and P for accepting or rejecting .Yf 0, and then observing additional observations would be a waste of time and effort. In other words, the possibility is raised that, by constructing the test in a fashion which permits termination of the sampling at any observation, one can test .Yf0 with fixed error sizes ct and P and yet do so with fewer than 64 observations on an average. This is in fact the case; although it may at first appear surprising in view of the fact that the best test for fixed sample size requires 64 observations. The saving in observations is often quite large, sometimes as much as 50 percent! We will study such a sequential procedure in the remaining subsections.

7.2

Definition of Sequential Probability Ratio Test

Consider testing a simple null hypothesis against a simple alternative hypothesis. In other words, suppose a sample can be drawn from one of two distributions (it is not known which one) and it is desired to test that the sample came from one distribution against the possibility that it came from the other. If Xl' X 2' . denotes the random variables, we want to test .Yf 0: Xi "-' lo( .) versus .Yf1: Xi "-' 11 (. ). The simple likelihood-ratio test was of the following form: Reject .Yf 0 if .< =

~: < k for some constant k > o.

The sequential test that we propose to consider employs the likelihood-ratios sequentially. Define

SEQUENTIAL TESTS OF HYPOTHESES

467

for m = 1, 2, ... , and compute sequentially )"h A2' ... ,. For fixed ko and kl satisfying 0 < ko < k1' adopt the following procedure: Take observation Xl and compute A1 ; if A1 s ko, reject :Yf 0; if A1 > kh accept :Yf0; and if ko < )"1 < k1' take observation X2, and compute A2 If A2 s ko, reject :Yf 0; if A2 > k1' accept :Yf 0; and if ko < A2 < kh observe X3' etc. The idea is to continue sampling as long as ko < A) < k1 and stop as soon as Am s ko or Am ~ k1' rejecting :Yf 0 if Am < ko and accepting :Yf 0 if Am > k 1 The critical region of the described sequential test can be defined as C =
n=1

U Cn , where

ao

)"n(Xh . , xn) S k o}.

(58)

A point in Cn indicates that :Yf 0 is to be rejected for a sample of size n. ilarly, the acceptance region can be defined as A
=
n= 1

Sim-

U An' where

ao

Definition 20 Sequential probability ratio test F or fixed 0 < ko < k1' a test as described above is defined to be a sequential probability ratio test. IIII

When we considered the simple likelihood-ratio test for fixed sample size n, we determined k so that the test would have preassigned size ct. We now want to determine ko and k1 so that the sequential probability ratio test will have preassigned ct and {J for its respective sizes of the Type I and Type II errors. Note that
ct = P[reject :Yf 0

1:Yf 0 is true] = I

00

n=1

f Lo(n)
en

(60)

and
{J = P[accept :Yf 0 1:Yf 0 is false] =

00

n=1

f L (n),
1

(61)

An

where, as before,

Sen Lo(n) is a shortened notation for J... J

en

[Ii

!o(Xi ) dXi]'

1=1

For fixed ct and {J, Eqs. (60) and (61) are two equations in the two unknowns ko and k 1 (Both An and Cn are defined in terms of ko and k 1 .) A solution of these two equations would give the sequential probability ratio test having the desired preassigned error sizes ct and {J. As might be anticipated, the actual

468

TESTS OF HYPOTHESES

IX

determination of ko and kl from Eqs. (60) and (61) can be a major computational project. In practice, they are seldom determined that way because a very simple and accurate approximation is available and is given in the next subsection. We note that the sample size of a sequential probability ratio test is a random variable. The procedure says to continue sampling until An = An(Xl' ... , xn) first falls outside the interval (ko, k l ). The actual sample size then depends on which Xi'S are observed; it is a function of the random variables Xl, X 2 , and consequently is itself a random variable. Denote it by N. Ideally we would like to know the distribution of N or at least the expectation of N. (The procedure, as defined, seemingly allows for the sampling to continue indefinitely, meaning that N could be infinite. Although we will not so prove, it can be shown that N is finite with probability 1.) One way of assessing the performance of the sequential probability ratio test would be to evaluate the expected sample size that is required under each hypothesis. The following theorem, given without proof (see Lehmann [16]), states that the sequential probability ratio test is an optimal test if performance is measured using expected sample size.

Theorem 10 The sequential probability ratio test with error sizes a and /3 minimizes both G[NI Jt o is true] and G[NI Jt l is true] among all tests (sequential or not) which satisfy the following: P[Jt 0 is rejected I Jt 0 is true] < a, P[Jt 0 is accepted I Jt 0 is false] < /3, and the expected sample size is finite. // //
Note that in particular the sequential probability ratio test requires fewer observations on the average than does the fixed-sample-size test that has the same error sizes. In Subsec. 7.4 we will evaluate the expected sample size for the example given in the introduction in which 64 observations were required for a fixed-sample-size test with preassigned a and /3.

7.3

Approximate Sequential Probability Ratio Test

We noted above that the determination of ko and kl that defines that particular sequential probability ratio test which has error sizes a and P is in general computationally quite difficult. The following remark gives an approximation to ko and k l

Remark Let ko and kl be defined so that the sequential probability ratio test corresponding to ko and kl has error sizes a and /3; then ko and kl can be approximated by, say, k~ and k~, where

k~

a
= 1-

/3

and

k~ =

1- a

(62)

SEQUENTIAL TESTS OF JIYPOTHESES

469

PROOF

(Assume

J,

P[N

= n IJt';] = 1 for i = 0,
n= 1 en

1.)

(1.

= P[reject -*' 0I-*'o is true] =

= ko

00

f Lo(n) < L f koL1(n)

00

n= 1

en

n= 1

L fenLl(n) =
00

ko P[reject

-*'0 I-*' 1 is true]

= k o(1- P),
and hence ko > (1.1( I - P). Also

1-

(1. = P[accept -*'0I-*'o is true)

= k1P[accept

-*'0 I-*' 1 is true] =

kIP,
k~ =

and hence kl < (1 - (1.)IP. and k~ = (1 - (1.)IP satisfy

Note that the approximations

(1.10 - P)

(1. ' k 0=1 _

P< k 0 < k 1 ~ 1 -P (1. =

k'

1-

(63)

1/11
Remark Let (1.' and P' be the error sizes of the sequential probability ratio test defined by leO and k't given in Eq. (62). Then (1.' + P' < (1. + p.
Let A' and C' (with corresponding A~ and C~) denote the acceptance and critical regions of the sequential probability ratio test . Then and k 1 defined by
PROOF

ko

and 1 - (1.'

f Lo(n) > 1-(1. f , L 1(n) = 1-(1. L P L P P'; n;;:; 1 An n 1 An

00 00
I

hence (1.'(1 - P) (1.(1 - P'), and (1 - (1.)P' < (l - (1.')P, which together implythat(1.'(1- P) + (1 - (1.)P' ~ (1.(1- P') + (1- (1.')por(1.' + P' < (1. + p.

///1

470

TESTS OF HYPOTHESES

IX

Naturally, one would prefer to use that sequential probability ratio test having the desired preassigned error sizes a and P; however, since it is difficult to find the ko and kl corresponding to such a sequential probability ratio test, instead one can use that sequential probability ratio test defined by ko and kl of Eq. (62) and be assured that the sum of the error sizes a' and P' is less than or equal to the sum of the desired error sizes a and p.

Approximate Expected Sample Size of Sequential Probability Ratio Test The procedure used in performing a sequential probability ratio test is to continue sampling as long as ko < Am < kl and stop sampling as soon as Am < ko or Am > k 1. If Zi = loge [!O(Xi)!.fl(Xi)], an equivalent test is given by the following:
Continue sampling as long as loge ko <
m
1

7.4

L Z i < loge kl' and stop sampling as soon

1

as

L Zi < loge ko (and then reject *'0) or L Zi > loge kl (and then accept *'0)'
1

As before, letN be the random variable denoting the sample size of the sequential probability ratio test, and let Zi = loge [fo(Xi)/Ji(Xi)]. Equation (64), given in the following theorem, is useful in finding an approximate expected sample size of the sequential probability ratio test.

Theorem 11 Wald's equation Let ZI, Z2, ... , Zn, ... be independent identically distributed random variables satisfying &[ IZi I] < 00. Let N be an integer-valued random variable whose value n depends only on the values of the first n Zi'S. Suppose &[N] < 00. Then &[ZI + ... + Z N] = &[N] . &[ZJ (64)
PROOF

&[ZI

+ ... + ZN] =
= = = = =

&[&[ZI

+ ... + ZNIN]]

L
n=1
00
00

00

&[ZI
n

+' ,,+ZnIN=n]P[N=n]

n=1 i=1

L L
00

&[ZIIN = n]P[N = n] &[Zd N

i= 1 n=

L Li
00
00

= n]P[N = n]

L
i== 1

&[Zi IN

> i]P[N > i]

L
i= 1

&[Zi]P[N > i]

= &[Z;1

i= 1

L P[N > i]

00

= &[Zi]&[N].

SEQUENTIAL TESTS OF HYPOTHESES

471

($[Zi]

= $[ZiI N > i]sincetheevent {N > i} depends only onZ1 , ""Z-1

and hence is independent of Z, . Also $ [N] = Eq. (6) of Chap. II.)

L P[N;;::: i] follows from

i= 1

00

II!!

If the sequential probability ratio test leads to rejection of J'e0' then the random variable ZI + ... + ZN < lo~ ko, but ZI + ... + ZN is close to lo&: ko since ZI + ... + ZN first became less than or equal to loge ko at the Nth observation; hence $[ZI + ... + ZN] ~ loge ko . Similarly, if the test leads to acceptance, $[ZI + ... + ZN] ~ logekl ;hence$[ZI + ... + ZN] ~ plogeko + (1 - p)lo&:kl' where p = P[ J'e 0 is rejected]. Using
_ 8[ZI + ... + ZN] [ $ N] $[Zi]
I'V

P lo&: ko + (l - p) loge kl
$[Zl]

I'V

we obtain
@

D[N I - v P '

.:n 0 IS

true

I'V I'V

rx loge ko + (I - rx) loge kl $[Zj I J'e0 IS true]

I'V

rx loge [rx!(1 - P)]

_________________

'"

- rx) loge [(1 - rx)!P] $[Zil J'eo is true] ,

~---------------

+ (1

_ __ _ _

(65)

and
$[N IJ'e0

.
IS

false] ~
,....,
I'V

(1 -

$[Z, I J'e0 is false]

P) loge ko + Ploge kl
+ P lo~ [(1
- rx)IP]
(66)

(l - P) loge [rxl(l - P)]

$[Zd J'e 0 is false]

EXAMPLE 27 Consider sampling from N(6, q2), where q2 is assumed known. Test J'e 0: 6 = 60 versus J'e 1 : 6 = 61 , Now

472 TESTS OF HYPOTHESES

IX

hence
8[Zd -*' 0 is true] = =

~ [(8~ 2(1
1

(1) - 280 (80

2

( 1)]

2(12

(8 1

( 0) ,

and

For a; = .01, P= .05, (12 Eq. (65) reduces to

100,80 = 100, and 8 1 = 105 (as in Example 26),

8J[NI1I',p'
({) en 0 IS

] '" .01 loge (.01/.95) + .99 loge (.99/.05) ,..., 24 true '" 25/200 '"
8[N I-*'o is false] ~ 34.

and The average sample sizes of 24 and 34 for the sequential probability ratio test compare to a sample size of 64 for the fixed-sample-size test. 1///

PROBLEMS
1

Let X have a Bernoulli distribution, where P[X = 1] = () = 1 - P[X = 0]. (a) For a random sample of size n = 10. test -*'0: () < i versus -*'1: () > i. Use the critical region {L x, 6}. (i) Find the power function, and sketch it. (ii) What is the size of this test? (b) For a random sample of size n 10: (i) Fmd the most powerful size-o: (0: .0547) test of -*'0: e= i versus

-*'1: () = 1.

1. (c) For a random sample of size 10, test -*'0: () i versus -*'1: e= 1. (i) Fmd the minimax test for the loss function 0 = t(do ; eo) = t(d1 ; ( 1 ), {(do; e .) = 1719, t(d.; eo) 2241. (ii) Compare the maximum risk of the minimax test with the maximum risk of the most powerful test given in part (b). (d) Again, for a sample of size 10, test -*'0: () 1 versus -*'1: = 1. Use the above loss function to find the Bayes test corresponding to prior probabilities given by
(ii) FIlld the power of the most powerful test at

= (1719/2241)0)10 + 34 '

PROBLEMS

473

Let X have the density f(x; 8) 8x'- 1 1(o.1)(X). (a) To test.:,y0: 8 < 1 versus Jf'1 : 8 > 1, a sample of size 2 was selected, and the critical region C {(Xl, X2): 3/4x. < X2} was used. Find the power function and size of this test. (b) For a random sample of size 2, find the most powerful size-[<x = 1(1 - In 2)] test of Jf' 0: 8 = 1 Vel\.:15 Jf'1: 8 2. (c) Are the tests that you ob~ined in parts (a) and (b) unbiased? (d) For a random sample of size 2, find the minimax test of Jf' 0: 80 = 1 versus Jf'1: 8. = 2 using the loss function {(do; 80 ) = {(d1 ; 81) 0, {(do; 81) = 1 - loge2, {(d1 ; 80 ) 1 + 108e2. (e) For a random sample of size n, find the Bayes test corresponding to prior probabilities given by g = i of Jf' 0: 8 = 1 versus Jf'1: 8 = 2 using the loss function {(do; 80 ) {(d1 ; 81) = 0, {(do; 81) = 1, {(d1 ; 80 ) = 2. (j) Test Jf' 0: 8 = 1 versus Jf'.: 8 = 2 using a sample of size 2. Let a. = size of Type I error and fJ size of Type II error. Find the test that minimizes the largest of <X and fJ. 3 Let = {l, 2}, and suppose you have one observation from the density 1(6-i-.6+i-)(x). Show that a test that has uniformly smallest risk among all tests exists, and find it. 4 Let X be a single observation from the density 2

f(x; 8)

8x'- 1 1(o.1)(x),

where 8 >0. (a) In testing Jf' 0: 8 <1 versus Jf'1: 8 > 1, find the power function and size of the test given by the following: Reject Jf' 0 if and only if X> 1. (b) Find a most powerful size-<x test of Jf' 0: 8 2 versus Jf'1: 8 = 1. (c) For the loss function given by {(do; 2) = {(d.; 1) 0, {(do; 1) = {(d1 ; 2) 1, find the minimax test of Jf' 0: 8 = 2 versus Jf'1: 8 = 1. (d) Is there a uniformly most powerful size-a. test of Jf' 0: 8 > 2 versus ~1: 8 < 2? If so, what is it? (e) Among all possible simple likelihood-ratio tests of Jf' 0: 8 = 2 versus Jf'1: 8 = 1, find that test that minimizes <X + fJ, where <X and fJ are the respective sizes of the Type I and Type II errors. (f) Find the generalized likelihood-ratio test of size <X of Jf' 0: 8 = 1 versus Jf'.:8#;1. 5 Let X be a single observation from the density f(x; 8) = (28x + 1 - 8)1[0. 11(X), where 1 <8<1. (a) Find the most powerful size-a. test of Jf' 0: 8 = 0 versus Jf'1: 8 = 1. (Your test should be expressed in terms of <x.) (b) To test Jf' 0: 8 <0 versus Jf'1: 8 0, the following procedure was used: Reject :?e0 if X exceeds 1. Find the power and size of this test. (c) Is there a uniformly most powerful size-a. test of Jf' 0: 8 <0 versus Jf' 1~ 8 > O? If so, what is it?

474

TESTS OF HYPOTHESES

IX

What is the generalized likelihood-ratio test of :f 0: 8 = 0 ve~us :f I! 8 O? Among all possible simple likelihood-ratio tests of :f 0: (J = 0 versus :f I: 8 = 1 find that test which minimizes 0'. + {3, where 0'. and (3 are the respective sizes of the Type I and Type II errors. (f) Given a set of observations. all of which fall between 0 and I, indicate how you would test the hypothesis that the ob~ ,~vations came from the density
(d) (e)

f(x; 8).

Let Xl, .. , Xn denote a random sample from f(x; 8) = (1/8)1(0. 6)(X), and let Y l , ... , Y n be the corresponding ordered sample. To test :f 0: 8 = 80 versus :f I: 8 80 , the following test was used: Accept :f 0 if 80 ( \Y;;) < Yn < 80 ; otherwise reject. (a) Find the power function for this test, and sketch it. (b) Find another (nonrandomized) test that has the same size as the given test, and show that the given test is more powerful (for all alternative 8) than the test you found. 7 Let Xl, . , Xn denote a random sample from
f(x; 8) =( 1/8)x(I-6)/61(0. l)(x).

Test :f 0: 8 < 80 versus :f I: 8> 80 (a) For a sample of size n, find a uniformly most powerful (UMP) size-O'. test if such exists. (b) Take n = 2, 8 0 = 1, and 0'. = .05, and sketch the power function of the UMP test. 8 Let Xl, .'., Xn be a random sample from the Poisson distribution e- 6 8J< f(x; 8) = -,-l{o.l. 2 )(x).

x.

(a)

Find the UMP test of :f 0: 8 = 80 versus :f I: 8 > 80 , and sketch the power function for 80 = 1 and n = 25. (Use the central-limit theorem. Pick
0'.

= .05.)

Test:f 0: 8 = 80 versus :fl: 8 80 Find the general form of the critical region corresponding to the test arrived at using the generalized likelihoodratio principle. (The critical region should be defined in terms of L Xl.) (c) A reasonable test of :f 0: 8 = 80 versus :f I: 8 80 would be the following: Reject if I X - 80 I >K. For 0'. = .05, find K so that P[reject :K 0 I:f 0] = .05. (Assume that n is large enough so that the central-limit theorem can be used to find an approximation to K.) 9 Let e = {80 , 81 }. Show that any test arrived at using the generalized likelihoodratio principle is equivalent to a simple likelihood-ratio test. 10 To test :f 0: 8 < 1 versus :f I : 8 > 1 on the basis of two observations, say Xl and X 2 , from the uniform distribution on (0, 8), the following test was used:
(b)

Reject:f 0

if

PROBLEMS

475

Find the power function of the above test, and note its size. [Recall that Xl + X 2 has a triangular distribution on (0, 28).] (b) Find another test that has the same size as the given test but has greater power for some () > 1 if such exists. If such does not exist, explain why. 11 Let Xl, . , Xn be a random sample of size n fromf(x; 8) = ()2 xe - 61cl(0. CO)(x). (a) In testing .1f 0: () < 1 versus .1f1 : () > 1 for n = 1 (a sample of size 1) the following test was used: Reject .1f 0 if and only if Xl < 1. Find the power function and size of this test. (b) Find a most powerful size-a test of .1f 0: () = 1 versus .1f1: () = 2. (c) Does there exist a uniformly most powerful size-a test of .1f 0: () < 1 versus .1f1: () > I? If so, what is it? (d) In testing .1f 0: () = 1 versus .1f 1: () = 2, among all simple likelihood-ratio tests find that test which minimizes the sum of the sizes of the Type I and Type II errors. You may take n = 1. 12 Let Xl, .. , Xn be a random sample from the uniform distribution over the interval (), () + 1). To test .1f 0: () = 0 versus .1f1: () > 0, the following test was used: Reject .1f 0 if and only if Yn > 1 or Y 1 > k, where k is a constant. (a) Determine k so that the test wiIl have size a. (b) Find the power function of the test you obtained in part (a). (c) Prove or disprove: If k is selected so that the test has size a, then the given test is uniformly most powerful of size a. 13 Let X .. . , Xm be a random sample from the density ()lxB 1 - 11(0.o(x), and let Y 1, "', Yn be a random sample from the density ()2y fJ2 -11(0, 1)(y). Assume that the samples are independent. Set Vf = -logeXf' i = 1, ... , m, and VJ = -loge YJ, j = 1, .. , n. (a) Find the generalized likelihood-ratio for testing .1f 0: ()1 = ()2 versus .1f 1 :()1()2. (b) Show that the generalized likelihood-ratio test can be expressed in terms of the statistic
(a)

L Vf T - =-....;;;;..;.----==--(c)

-LU,+LV/

If.1f 0 is true, what is the distribution of T? (You do not have to derive it if you know the answer.) Does the distribution of T depend on () = ()1 = ()2 given that .1f 0 is true? 14 Find a genera1ized likelihood-ratio test of size a: for testing .1f0: () < 1 versus .1f 1: () > 1 on the basis of a random sample Xl, "', Xn from f(x; 8) = ()e- b l(o. CO)(x). 15 Let X be a single observation from the density f(x; 8) = (1 + 8)xBl(o. 1)(x), where (-l. (a) Find the most powerful size-a: test of .1f 0: () = 0 versus .1f 1: () = 1. (b) Is there a uniformly most powerful size-a: test of.1'f'0 : () < 0 versus .1f1 : () > O? If so, what is it?

476

TESTS OF HYPOTHESES

IX

16

17

18

19

20

21

22

23 24

25

Among all possible simple likelihood-ratio tests of :K 0: () 0 versus :K.: () = 1 find a test which minimizes 2/X + p, where /X and p are the respective sizes of the Type I and Type II errors. (d) Find a generalized likelihood-ratio test of :K 0: () = 0 versus :K 1: () if; 0.. Let X., , Xm bea random sample from ().e- 6,J(I(o. CCI)(X), and let Y., , Yn be a random sample from ()2 e- 6271(0. CCI)(y)' Assume that the samples are independent. (a) Find the generalized likelihood-ratio for testing :K 0: ()1 = ()2 versus :K.: ()l =f; ()2 (b) Show that the generalized likelihood-ratio test can be expressed in terms of the statistic T = .2 X,f(.2 X, + .2 YJ). Argue (or show) that the distribution of T does not depend on () = (). = ()2 when :K 0 is true. Use the confidence-interval technique to derive a test of :K 0: P-l = P-2 versus :K 1: p-. =f; P-2 in sampling from the bivariate normal distribution. Such a test is often called a paired t test. (See the last paragraph in Subsec. 3.4 in Chap. VIII.) Given the sample (- .2, - .9, - .6, .1) from a normal population with unit variance, test whether the population mean is less than 0 at the .05 level (Le., with probability .05 of a Type I error). That is, test :K 0: p- 0 at the .05 level versus :K. : p- > O. Given the sample (-4.4, 4.0, 2.0, - 4.8) from a normal population with variance 4 and the sample (6.0, 1.0, 3.2, - .4) from a normal population with variance 5, test at the .05 level that the means differ by no more than one unit. Plot the power function for this test. Plot the ideal power function. A metallurgist made four determinations of the melting point of manganese: 1269, 1271, 1263, and 1265 degrees centigrade. Test the hypothesis that the mean II. of this population is within 5 degrees centigrade of the published value of 1260. Use /X = .05. (Assume normality and (12 5.) Plot the power function for a test of the null hypothesis :K 0: - 1 < p- < 1 for a normal distribution with known variance using sample sizes 1, 4, 16, and 64. (Use the standard deviation (J as the unit of measurement on the fL axis and .05 probability of Type I error.) Plot the ideal power function. Let Xl, ... , Xn be a random sample of size n from a normal density with known variance. What is the best critical region for testing the null hypothesis that the mean is 6 against the alternative that the mean is 4 ? Derive a test of :K 0: (J2 < 10 against :K1: (J2 > 10 for a sample of size n from a normal population with a mean of O. In testing between two values 11.0 and fLl for the mean of a normal population, show that the probabilities for both types of error can be made arbitrarily small by taking a sufficiently large sample. A cigarette manufacturer sent each of two laboratories presumably identical samples of tobacco. Each made five determinations of the nicotine content in milligrams as follows: (i) 24, 27, 26, 21, and 24 and (ii) 27, 28, 23, 31, and 26. Were the two laboratories measuring the same thing? (Assume normality and a common variance.)
(c)

PROBLEMS

477

26

27 28

29
30

31
32

33

34

35
36

37

The metallurgist of Prob. 20, after assessing the magnitude of the various errors that might accrue in his experimental technique, decided that his measurements should have a standard deviation of 2 degrees centigrade or less. Are the data consistent with this supposition at the .05 level? (That is, test .:f0: a <2.) Test the hypothesis that the two samples of Prob. 19 came from populations with the same variance. Use Ct = .05. The power function for a test that the means of two normal populations are equal depends on the values of the two means ILl and 1'2 and is therefore a surface. But the value of the function depends only on the difference (1 = 1'1 - 1'2, so that it can be adequately represented by a curve, say {3(U). Plot {3(U) when samples of 4 are drawn from one population with variance 2 and samples of 2 are drawn from another population with variance 3 for tests at the .01 level. Given the samples (1.8, 2.9, 1.4, 1.1) and (5.0, 8.6,9.2) from normal populations, test whether the variances are equal at the .05 level. Given a sample of size 100 with X = 2.7 and 2: (X, X)2 225, test the null hypothesis .:f 0: IL 3 and a 2 = 2.5 at the .01 level, assuming that the population is normal. Using the sample of Prob. 30, test the hypothesis that I' a 2 at the .01 level. Using the sample of Prob. 30, test at the 0.1 level whether the .95 quantile point, say ~ = ~.95, of the population distribution is 3 relative to alternatives ~ < 3. Recall that ~ is such that J~ ~ f(x) dx = .95, where f(x) is the population density; it is, of course, I' + I.645a in the present instance where the distribution is assumed to be normal. A sample of size n is drawn from each of k normal populations with the same variance. Derive the generalized likelihood-ratio test for testing the hypothesis that the means are all O. Show that the test is a function of a ratio which has the F distribution. Derive the generalized likelihood-ratio test for testing whether the correlation of a bivariate normal distribution is O. If XI, X 2 , , XII are observations from normal populations with known variances ai, vi, ... , a~, how would one test whether th~ir means were all equal? A newspaper in a certain city observed that driving conditions were much improved in the city because the number of fatal automobile accidents in the past year was 9 whereas the average number per year over the past several years was 15. Is it possible that conditions were more hazardous than before? Assume that the number of accidents in a given year has a Poisson distribution. Six 1-foot specimens of insulated wire were tested at high voltage for weak spots in the insulation. The numbers of such weak: spots were found to be 2, 0, 1, 1, 3, and 2. The manufacturer's quality standard states that there are less than 120 such defects per 100 feet. Is the batch from which these specimens were taken worse than the standard at the .05 level ? (Use the Poisson distribution.)

478

TESTS OF HYPOTHESES

IX

38 Consider sampling from the normal distribution with unknown mean and variance: (a) Find a generalized likelihood-ratio test of :K 0: u 2 < u~ versus :K I: u 2 > u& (b) Find a generalized likelihood-ratio test of :K 0: u 2 u6 versus :K I: u 2 :f:. u~. 39 (a) Suppose (Nit 0, Nt) is multinomially distributed with parameters n,
o.

PI, .. ,p", where NUl

= n- N.

~.,.

N"andp"+l = 1

" 2PJo Theorem

1

8 states that

(b)

has a limiting chi-square distribution. Find the exact mean and variance of Q. Let (Nt, N,,) be distributed as in part (a). Define
0 .,

[See Eq. (25).] Find 8[Qf]. [See Eq. (26).] Is 8[Q~] for PI = p~, .. , PUI = P~+l less than or equal to 8[Qf] for arbitrary Ph ... , p,,+!? 40 A psychiatrist newly employed by a medical clinic remarked at a staff meeting that about 40 percent of all chronic headaches were of the psychosomatic variety. His disbelieving colleagues mixed some pills of plain flour and water, giving them to all such patients on the clinic's rolls with the story that they were a new headache remedy and asking for comments. When the comments were all in they could be fairly accurately classified as follows: (i) better than aspirin, 8, (ti) about the same as aspirin, 3, (iii) slower than aspirin, 1, and (iv) worthless, 29. While the doctors were somewhat surprised by these results, they nevertheless accused the psychiatrist of exaggeration. Did they have good grounds? 41 A die was cast 300 times with the following results: Occurrence: Frequency:

1 43

2 49

3 56

4 45

5
66

6 41

Are the data consistent at the ,05 level with the hypothesis that the die is true? 42 Of 64 offspring of a certain cross between guinea pigs, 34 were red, 10 were black, and 20 were white. According to the genetic model, these numbers should be in the ratio 9/3/4. Are the data consistent with the model at the .05 level? 43 A prominent baseball player's batting average dropped from .313 in one year to .280 in the following year. He was at bat 374 times during the first year and 268 times during the second. Is the hypothesis tenable at the .05 level that his hitting ability was the same during the two years? 44 Using the data of Prob. 43, assume that one has a sample of 374 from one Bernoulli population and 268 from another. Derive the generalized likelihood-ratio test for testing whether the probability of a hit is the same for the two populations. How does this test compare with the ordinarY test for a 2 x 2 contingency table?

PROBLEMS

479

45 The progeny of a certain mating were classified by a physical attribute into three groups, the numbers being 10 53, and 46. According to a genetic model the frequencies should be in theratios p 2j2p{1-p)j{l-p)2. Are the data consistent with the model at the .05 level? 46 A thousand individuals were classified according to sex and according to whether or not they were color-blind as follows:
9

Male

Female

Normal Color-blind

442 38

514
6

According to the genetic model these numbers should have relative frequencies given by
P 2

p2 -+pq 2 q2 2

q 2

where q = 1 - p is the proportion of color-blind individuals in the population. Are the data consistent with the model ? 47 Treating the table of Prob. 46 as a 2 x 2 contingency table, test the hypothesis that color blindness is independent of sex. 48 Gilby classified 1725 school children according to intelligence and apparent family economic level. A condensed classification follows:
Dull Intelligent
Very weB c10thed Well c10thed Poorly c10thed 81

Very capable
233

322

141 127

457 163

153
48

Test for independence at the .01 level. 49 A serum supposed to have some effect in preventing colds was tested on 500 individuals, and their records for 1 year were compared with the records of 500 untreated individuals as follows:
More than one cold
103 140

No colds
Treated Untreated 252

One cold

145
136

224

Test at the .05 level whether the t~o trinomial populations may be regarded as the same.

480 TESTS OF HYPOTHESES

IX

50

According to the genetic model the proportion of individuals having the four blood types should be given by:
0: q2 A:p2+2pq B: r2 2qr AB: 2pr

51

where p + q + r = I. Given the sample 0, 374; A, 436; B, 132; AB, 58; how would you test the correctness of the model? Galton investigated 78 families, classifying children according to whether or not they were light-eyed, whether or not they had a light-eyed parent, and whether or not they had a light-eyed grandparent. The following 2 x 2 x 2 table resulted :
Grandparent Light Parent Ught Not Light Not Not

:a U
52

'0

Ught Not

1928 303

552 395

596 225

508 501

53

54

55

Test for complete independence at the .01 level. Test whether the child classification is independent of the other two classifications at the .01 level. Compute the exact distribution of A for a 2 x 2 contingency table with marginal totals Nt. = 4, N 2. = 7, N.1 = 6, N.2 = 5. What is the exact probability that - 21o&e A exceeds 3.84, the .05 level of a chi-square distribution for one degree of freedom? In testing independence in a 2 x 2 contingency table, find the exact distribution of the generalized likelihood-ratio for a sample of size 2. Do the same for samples of size 3 and 4. Discuss. Let Xl, . , Xn be a random sample from N(I-', a 2 ), where a 2 is known. Let A denote the generalized likelihood-ratio for testing Jf' 0: I-' = 1-'0 versus :Yf 1: I-' :f:. 1-'0' Find the exact distribution of - 2 lo&eA, and compare it with the corresponding asymptotic distribution when Jf' 0 is true. HINT: L (X, - X)2 = '" - - 1-') 2 L. (X, - I-')2 - n(X Here is an actual sequence of outcomes for independent Bernoulli trials. Do you think p (the probability of success) equals l?

s I I I s, sIs I I, s I I I I, I I sIs, s I I I I, s I Is/, I I I I s, I I s I /, s I I I I, s s I I I.

PROBLEMS

481

~.
If you do not think p is 1, what do you think p is? Give a confidence-interval estimate of p. If the above data were generated by tossing two dice, then what would you think p is? If the data were generated by tossing two coins, then what would you think pis? (If the data were generated by tossing two dice, assume that the possible values of pare j/36, j 0, ... , 36. If the data were generated by tossing two coins, assume that the possible values of p arej/4,j = 0, ... ,4.) 56 In sampling from a Bernoulli distribution, test the null hypothesis that p = t against the alternative that p = t. Let p refer to the probability of two heads when tossing two coins, and carry through the test by tossing two coins, using (X = f3 .10. (The alternative was obtained by reasoning that tossing two coins can result in the three outcomes: two heads, two tails, or one head and one tail, and then assuming each of the three outcomes equally likely.) 57 Show that the SPRT (sequential probability ratio test) of f' = fLo versus f' = fLl for the mean of the normal distribution with known variance may be performed by plotting the two lines

and

in the ny plane and then plotting

2:
1

II

Xi against n as the observations are made.

The test ends when one of the lines is crossed. 58 Consider sampling from f(x; e) = (1/9)1(0. B)(X), e> o. Discuss the sequential versus () = ()1 with ()o < ()1. probability ratio test of 59 Let XI, X 2 , " ' , XII' .. be independent random variables all having the same Bernoulli distribution given by P[XII = l] = () = 1 - P[XII = 0]. To test -*'0: e t versus -*'1: () = 1, the following sequential test was used: Continue sampling as long as n/2 - 2 < 2: Xi < n/2 + 2; if and when 2: Xi is first less than or equal to n/2 - 2, accept -*'0; and if and when 2: Xi is first greater than or equal to n/2 + 2, accept -*'1' Is this test a SPRT 1 60 Assume that X has a Poisson distribution with mean Consider testing -*' 0: () = I versus -*'1: () 2. Fix (X = f3 = .05. (a) Find the fixed sample size necessary to achieve the prescribed error sizes. (b) Derive the (approximate) sequential probability ratio test, and show that it

e eo

e.

" XI, n = 1, 2, ... can be based on the statistics 2:

i-I

(c)

Find the approximate expected sample sizes for the sequential probability ratio test.

x
LINEAR MODELS

1 INTRODUCTION AND SUMMARY

The purpose of this chapter is to discuss a special case of the linear statistical model. There is a large amount of material available on this subject, but we will only discuss the special case of the simple linear model. Some authors refer to this as the theory of straight-line regression. To study this model will require the use of some of the theory in previous chapters, such as distribution theory, point and interval estimation concepts, and some material from hypothesis testing. This chapter will demonstrate how these concepts can be utilized for a situation, the simple linear model, that is important in applied statistics. In Sec. 2 two examples are given to illustrate how the simple linear model can be used to simulate real-world problems. In Sec. 3 the simple linear model will be rigorously defined and put into a framework that will allow us to study it by using s.tatistical procedures from previous chapters. In the remaining sections discussion will be centered around point estimation, interval estimation, and testing hypotheses on the parameters in the model under two different assumptions about the distributions of the random variables in the model.

EXAMPLES OF THE LINEAR MODEL

483

2 EXAMPLES OF THE LINEAR MODEL

In this section we will give two examples to illustrate how the linear model arises in applied problems.

EXAMPLE 1 The distance s that a particle travels in time t is given by the formula s = Po + Plt, where Pl is the average speed and Po is the position at time t = O. If Po and Pl are unknown, then s can be observed for two distinct values of t and the resulting two equations solved for Po and Pl' For example, suppose that s is observed to be 2 when t = 1, and s is 11 when t = 4. This gives 2 = Po + Pl and 11 = Po + 4Ph and the solution is Po = -1, Pl = 3; so s = -1 + 3t. Suppose that for some reason the distance cannot be observed accurately, but there is a measurement error which is of a random nature. Therefore s cannot be observed, but suppose that We can observe Y, where Y = s + E and E is a random error whose mean is O. Substituting for s gives us

where Yis an observable random variable, t is an observable nonrandom variable, Eis an unobservable random variable, and Po and Pl are unknown parameters. We cannot solve for Po and Pl by observing two sets of values of Yand t, as We did with sand t above, since there is no functional relationship between Yand t. The objective in this model is to find Po and Pl and hence evaluate s = Po + Pl t for varipus values of t. Since s is subject to errors and cannot be observed, We cannot know Po and Pl, but by observing various sets of Yand t values statistical methods can be used to obtain estimates of Po, Pl' and s. This type of model is a functional-relationship model with a measurement error. / / //

EXAMPLE 2 For another example, consider the relationship between the height h and weight W of individuals in a certain city. Certainly there is no functional relationship between wand h, but there does seem to be some kind of relation. We shall consider them as random variables and shall postulate that (W, H) has a bivariate normal distribution. Then the expected value of H for a given value w of W is given by
8[HI W

= w] =

Po + P1W,

(2)

where Po and Pl are functions of the parameters in a bivariate normal density. Although there is no functional relationship between Hand W, if

484

LINEAR MODELS

they are assumed jointly normal, there is a linear functional relationship between the weights and the average value of the heights. Thus we can write the following: Hand Ware jointly normal, and
8[HI W = w] = Po

+ PI W ;

or We can write
H w = Po

+ PI W + E,

where E is a normally distributed random variable denoting error. This is a regression model, and although it came from a somewhat different problem than the functional relationship in Example 1, they both are special cases of a linear statistical model, which will be discussed in this
~~6.

3 DEFINITION OF LINEAR MODEL

Let Jl( . ) be a linear function of a real variable x. This is defined by Jl(x) = Po + PIX, where x is in a domain D. Quite often D will be the entire real line, a half line, or a bounded interval on the real line. To model the situations referred to in Examples 1 and 2 above, we assume that there exists a family of c.d.f.'s (one c.d.f. for each x in D) such that the mean of the c.d.f. corresponding to a given x (say xo) in D is Po + PIXO' Thus the means of the c.d.f.'s are on the line defined by Jl(x) = Po + PIX., See Fig. 1. The objective is to sample some of the c.d.f/s and on the basis of the sample to make statistical inferences about Po, Ph etc. The sampling is accomplished as follows:
(i) A set of n x's in D is observed and denoted by Xh X2' ... , X n. The x's are not random variables, but they may be selected either by

some random procedure or by purposeful selection. (ii) Each Xi determines a c.dJ. whose mean is Po + PIX! and whose variance is a 2 From this c.d.f. a value is selected at random and denoted by Y,. (Y 1 is a shortened notation for YXt ' ) Thus We have a set ofn pairs of observations, which we denote by (Yh Xl), (Y2 , X2), ... , (Yn , Xn). We have assumed that

and

DEFINITION OF LINEAR MODEL

485

~~----~--------~------~--~x

FIGURE 1

Therefore we can define random variables E l , E 2 ,

En by

for i = 1, 2, ... , n, and the Ei satisfy

and

So we can write for i = I, 2, ... , n, where and and this defines a linear model. We summarize these ideas below.

Definition 1 Linear model Let the function J1('} be defined by J1(x} = Po + PIX for all x in a set D. For each x in Diet F y",( . } be' a c.d.f. with a mean equal to tt(x), that is, Po + PIX, and variance 0'2. Let Xl> X2' "" xn bean observed set ofnx's from D. For Xi let Yj be a random sample of size 1 from thec.d.f. F yxt (' ) for i = 1,2, ... , n. Then (Yl , Xl), (Y2 , X2), . , (~, xn) is a set of n observations related by

486

LINEAR MODELS

and var [Yi ] = (12, i = 1,2, ... , n. These specifications define a linear statistical model.

(3)

IIII

Note We can write Eq. (3) as

Yj = /30 + /31 X, + Ei G[Ei ] = 0 var [Ei] = (12, wherei
=

(4)

1,2, ... , n.

/1/1

Note The word" linear" in "linear statistical model " refers to the fact that the function p(.) is linear in the unknown parameters. In the simple example We have referred to, p( . ) is defined by p(x) = /30 + /31 x; x in D, and this is linear in x, but this is not an essential part of the definition of this linear model. For example, Y = p(x) + E, where p(x) = /30 + /31 ~ is a linear statistical model. IIII Note In many situations some additional assumptions on the c.dJ. Fyx( ) will be made, such as normality. Also, generally the sampling procedure will be such that the Yi will be either jointly independent or pairwise uncorrelated. In fact We shall discuss inference procedures for two sets of assumptions on the random variables defined in Cases A and B 1/1/ below. Case A For this caSe We assume that the n random variables are jointly IIII independent and each Yi is a normal random variable. Case B For this case We assume only that the Yi are pairwise uncorrelated; that is, cov [Yj, YJ] = 0 for all i :F j = I, 2, ... , n. 1//1
For Case A We shall discuss the following:
(i) Point estimation of /30, /31, (12, and p(x) for any x in D (ii) Confidence interval for /30' /31, (12, and p(x) for any x in D (iii) Tests of hypotheses on /30, /31' and (12

For Case B We shall discuss the following: (iv) Point estimation of /30' /31, (12, and p(x) for any x in D

POINT ESTIMATION-CASE A

487

POINT ESTIMATION-CASE A

For this case Y I , Y2, ... , Yn are independent normal random variables with means Po + PIXh Po + PI X2, . , Po + PIX,. and variances a2 To find point estimators, We shall use the method of maxim urn likelihood. The likelihood function is

(5)

and log L(Po , PI' a ) =

2

n n 2 1~ -2 log 21t - -2 log a - 2 2 1... (Y t a i=l

Po - PtXi) .

The partial derivatives of log L(Po, Ph a 2 ) with respect to Po, Ph and a 2 are obtained and' set equal to O. We let Po, PI' 8 2 denote the solutions of the resulting three equations. The three equations are given below (with some minor simplifications):

(6)

The first two equations are called the normal equations for determining Pl' They are linear in Po and PI and are readily solved. We obtain

Po and
(7) (8) (9)

These are maximum-likelihood estimates of Ph Po, and a 2 , respectively. We notice that the x/s must be such that L (Xi - X)2 -=J:. 0; that is, there must be at least two distinct values for the Xi'

488

LINEAR MODELS

Note that since

fr.(Yi; PO' P1> 0-) =

(21ta'r t exp [ -! (Y'

- P:- P,

Xr]

= (21tC1

rt eXp [ -

2~2 (Po + P,xii]

Pl ) + Po 0'2 Yi + 0'2 Xi Y i ,

1 2 exp ( - 20'2 Y i

!Yj(Yi; Po, Ph 0') is a member of a three-parameter exponential family; hence, by a generalization of Theorem 16 of Chap. VII
i= 1

L Yi,

i= ]

"X L. , y.

(10)

is a set of minimal sufficient and jointly complete statistics. Furthermore, since the set of statistics given in Eq. (10) is a one-to-one transformation of the estimators (statistics) defined by Eqs. (7) to (9), the estimators are themselves minimal sufficient and jointly complete. To further examine the properties that the estimators possess, we shall find the joint distribution of statistics corresponding to Po, Pb 0'2. To do this, we shall first find the moment generating function of 0 1 , O2 , and 0 3 , which are random quantities with values defined by
1\ _

ttl -

Po - Po , 0'

1\

tt2 -

p] - p] ,
0'

(11)

By Definition 25 of Chap. IV the joint moment generating function of 0 h O2 , 0 3 is defined to be met 1, t 2 , t 3 ) = G[i 101 +t202 +130 3 ] if the expectation exists for - h < ti < h for some h > O. We obtain

exp [ -

hL

(Yi -

Po - p, x,i ]
dYl ... dYn'

(21t0'2)"/2

where in the integral the quantities ~1' ~2' ~3 will be written in terms of Yi and Xi' This integral is straightforward but tedious to evaluate, and the result is

L m(t 1 ,t 2 ,t3 )= ( expt ( t 2 1 ,,(

+ t2
2

L. Xi - X)

xNn _2+2tlt2 [-"C x

L. Xi -

-2 X)

L (Xi 1_

x)2

}) x (1 - 2t

)-(n-2)/2
3

'3

for t <to

POINT ESTIMATION-CASE A

489

From this moment generating function we can learn a number of things:

It factors into a function of tl and t2 only times a function of t3 only_ We write this result as m(tb t 2 , t 3 ) = ml(tb t2)m2(t 3 )- By using Theorem 10 of Chap. IV we know that the random variables associated with t I and t2 are independent of the random variable associated with
(i)

t3; that is, 0 1 and O2 are independent of0 3 , which implies that the maximum-likelihood estimators of /30 and /31 are jointly independent of the maximum-likelihood estimator of 0'2_ (ii) Since by a generalization of Theorem 7 in Chap. II a moment generating function uniquely determines the distribution of the random variables involved, we shall try to recognize the form of ml(t l , t2 ) and the form of m2(t 3 ) We note by Theorem 12 of Chap. IV that ml(t 1, t 2) is the moment generating function of a bivariate normal distribution, and, of course, we obtain the means, variances, and covariance. We see that the random variables, say Do and D., associated with Po and P1 are bivariate normal random variables with means (/30' /31) and covariance matrix
0'2
n

L (Xl -

L xt
X)2

_O'2;X

L (Xi 0'2

X)2
(12)
X)2

_O'2j

L (Xi -

j)2

L (Xi -

Another way to state this is the following: random variable with parameters

(Do, DI ) is a bivariate no.rmal

and cov

[Do, Dd = L (Xl - X) -2

-u2 x

(iii) We recognize that m2(t 3) is the moment generating function of a chi-square random variable with n - 2 degrees of freedom. Hence we have

490

UNEAR MODELS

which is distributed as a chi-square distribution with n - 2 degrees of freedom. (Here, and in the rest of this chapter, &2 and a2 are used to denote the random variables with values lJ2 and ij2 respectively.) By Eq. (22) of Chap. VI we get

na ] tff [ (12
so we define lJ2 by
A2
(I

=n-2;

-2
(I

n-2

1 f 2 2 i...J (Yi - Po - PI X,) . n- i=l

A A

We shall summarize these results in the following theorem.

Theorem 1 Consider Case A of the simple linear model given in Definition 1. The maximum-likelihood estimators of PI' Po, and (12 (corrected for bias) are given by

n _I
1 -

(Yi

Y)( Xi

x)

(Xi -

xl

'
(13)

These estimators satisfy the following: They are jointly complete sufficient statistics. (ij) They are unbiased estimators of their respective parameters. (iii) (no, n I) is independent of &2. (iv) (no, BI) has a bivariate normal distribution with mean (Po, PI) and covariance matrix given by Eq. (12). (v) (n - 2)it2/(l2 is a chi-square random variable with n - 2 / // / degrees of freedom.
(i)

In Chap. VII, we noted that maximum-likelihood estimators possess a number of good properties, but they, in general, are not minimum-variance unbiased estimators. We now employ a minor generalization of Theorem 11 of Chap. VII along with the results of Theorem 1 above, to state a strong optimal property about the estimators no, n h &2 of Po, Ph (12.

Theorem 2 Consider the simple linear model given in Definition 1. Let 'C(Po, Ph (12) be any known function of the parameters Po, Ph and (12 for which an unbiased estimator exists. Then there exists an unbiased

CONFIDENCE INTERV AL8-CASE A

491

estimator of 't(PO' PI' 0'2) that is a function of h and &2. We denote this estimator by o , h &2), and it is the UMVUE of 't(Po, Ph 0'2).

I(n n no, n

no, n

This result follows from a generalization of Theorem 17 of Chap. VII, since l , &2 is a set of sufficient complete statistics. 11//
PROOF

Corollary The UMVUE of each of the parameters Po, PI' and 0'2 is 1III given by h and &2, respectively, in Theorem 1.

no, n

Corollary The UMVUE of Jl(x) = Po + PIX for any X in the domain D is flex), where p(x) = + 1x. (fl(x) is the random variable with IIII values Jl(x) = Po + PIX.)

no n

Corollary For any two known constants c1 and c2 the UMVUE of c1Po + C2 /31 is CI +C 2 l 1//1

no

CONFIDENCE INTERVALS-CASE A
_ {n - 2)&2 U ---al-=---

To obtain a y-Ievel confidence interval on al, we note by Theorem I that

is distributed as a chi-square random variable with n - 2 dJ. (degrees of freedom). Hence U is a pivotal quantity, and we get
P[Xtl -y)/2(n
-

2) ::; U ::; Xtl +y)/2(n

~2 cr

2)] = y.

If we substitute for U and simplify, we get

- 2)&2 p[X(l(n +Y)/2(n - 2)
2

<

(n - 2)&2
2

X(I-y)/2(n - 2)

= 1',

(14)

and this is a 1001' percent confidence interval on 0'2. To obtain a y-Ievel confidence interval on Po, we note that by Theorem 1:
(i) Z = Po)J'L (Xi - x)2n10'2 xf is distributed as a standard normal random variable. (U) (n - 2)&210'2 = U is distributed as a chi-square random variable with n - 2 d.f. (iii) Z and U are independent.

(no -

Hence, by Theorem 10 of Chap. VI

492

LINEAR MODELS

is distributed as Student's t distribution with n - 2 d.f. quantity. We get

P[ t(1 +'1)/2(n -

Hence T is a pivotal
2)]

2) ~ T <

t(1 +'1)/in -

= y,

and if we substitute for T, we get

{ -t(l+,)/.(n - 2) <

J}o; Po In Lr~f x)' < t(1+,)/.(n - 2)]

= y.

Mter simplifying We get the following for a l00y percent confidence interval on Po:
p no -

t(I+'1)/2(n - 2)&

L (Xi -'2 n X)
-

Lxf

< Po < no + t(1+1)/2(n

We note that by Theorem 1 we get

2)&

] LLX; (Xi - xi = y.

and the estimated variance of no, which we write as var [no], is given by
,... r8] var LllO
=

~2
0-

LX;
n L..
" (

Xi -

-)2 X

Then the confidence statement can be written as

p[no -

t(1+'1)/2(n -

2)Jvar [no]

Po < no

+ t(I+1)/2(n -

2)Jvar [no]] -

y.
(15)

To obtain a y-Ievel confidence interval on

pb we note that by Theorem 1:

2 (Xi - x)2ja is distributed as a standard (i) Z = (n 1 normal random variable. (ii) U = (n - 2)&2ja2 is distributed as a chi-square random variable with n - 2 d.f. (iii) Z and U are independent.

pl)JL

Hence by Theorem 10 of Chap. VI

CONFIDENCE INTERVALS-CASE A

493

is distributed as Student's t distribution with n - 2 dJ. quantity, and we get

P[ t(1 +y)/2(n -

Hence T is a pivotal 2)] = 1'.

2) < T <

t(J +y)/2(n -

If we substitute for T, we obtain

{ - t(1+Yl/2(n -

2)

~ (n, - P,)J'I (X~-; X)2 < t(1 +Yl/z(n -

2)J

= y.

After some simplification we get

p[fJ] -

t(1 +y)/2(n -

2)J'I (Xi1t-

X)

_ 2

<p, <

jj,

+ t(1 +Yl/2(n -

2)J'I (X~~ J
X)2

y,

and this is a 1001' percent confidence interval on that

/31'

We note from Theorem 1

(12

var

[fJ d = L (Xi

X)

-2

and that the estimated variance of fJ], which is denoted by var [8]], is given by var

[fJ d = L (Xi

a2
-

-)2 .

If we substitute this into the confidence-interval statement, we obtain

p[fJ 1 -

t(1

+ y)/i n -

2)J~ar [fJd < /31 < fJ]

+ t(l +y)/2(n -

2)Jvar

[fJ1 J] = 1'.
(16)

To obtain a y-ievel confidence interval on J.l(x) for any we note that (i) J.l(x) = /30 + /3]x. (ii) fi(x) = fJo + fJ 1 (X). (iii) G[fi(x)] = J.l(x). (iv) var [fi(x)] = var [8 0 + fJ] x] = var [fJo] + 2xcov [fJo, fJ]]

in the domain D,

=
=

L (Xi 2

(12

X)2

(L x~ _ 2xx + X2)
n

+ x 2 var [fJd

'I(x:~ x)2 [(X - x)' + ~ 'I (x, -

X)2]

= (1

[1~ + L(x (Xi

X)2 ] - X)2 .

494

UNEAR MODELS

(v) Z = [tl(X) - Jl(x)]/Jvar [jl(x)] is distributed as a standard normal random variable. (vi) U = (n - 2)&2/q2 is distributed as a chi-square random variable with n - 2 d.f. (vii) U and Z are independent. (viii) T
=

tl(x) - Jl(x) =~fio var[ft(x)] J8'2[I/n

+ filx - Po - PIX + (x - x)2tf. (Xi - xi]

Hence T is a

is distributed as Student's t distribution with n - 2 d.f. pivotal quantity, and we obtain

P[ -t(1+1)/2(n - 2) ~ T ~ t(1+1)/2(n - 2)] = y.

If we substitute for T and simplify, we get

or
p[fio

+ fi 1x - t(1+1)/2(n - 2)Jvar [tl(x)] ~ Po + P1 X

~

Do + fi 1x + t(1 +1)/2(n -

2)Jvar [(l(x)l]

1',

and a 1001' percent confidence interval for Po

+ P1X is obtained.

6 TESTS OF HYPOTHESES-CASE A
In the linear model there are many tests that could be of interest to an investigator. For example, he may want to test whether the line goes through the origin, i.e., to test if the intercept is equal to zero, or perhaps test whether the intercept is positive (or negative). These are indicated by
Jf0: Po = 0 versus Jf1 : Po i= 0, Jf0: Po > 0 versus Jf1 : Po < 0, Jf0: Po < 0 versus Jf1 : Po >

o.

These tests indicate that there is no interest in the slope P1 or the variance q2

TESTS OF HYPOTHESES-<::ASE A

495

On the other hand the interest may be in the slope rather than the intercept, and an investigator could be interested in testing

-*' 0: PI = 0 versus-*, 1: PI -*' 0=

-::f:.

0,

P1 < 0 versus -*'1: P1 > 0,

etc. Rather than testing whether the intercept (or slope) is equal to 0 an investigator may be interested in testing whether it is equal to a given number. For exam pIe he may be interested in testing

-*'0: Po = 2 versus -*'1: Po < 2, -*'0: PI = 1 versus -*'1 : P 1 -::f:. 1,

etc. We shall derive a test of the hypothesis

We could just as well derive a test of the hypothesis

or a test of the hypothesis

To test

one obvious choice for a test statistic is

Under-*,0 the random variable T is distributed as Student's t distribution with n - 2 degrees of freedom. Thus a test procedure with size t:J. is the following: Reject-*, 0 if and only if IT I > t 1 _ a/ 2 (n - 2). By comparing this with Eq. (16) we notice that this test is equivalent to the procedure of setting a 1 - t:J. confidence interval on the parameter PI and rejecting the hypothesis if and only if the confidence interval does not contain O. We will now show that this test is a generalized likelihood-ratio test. Corresponding to the notation in Chap. IX we note that in testing

-*'0: PI

0 versus -*' 1: PI

-::f:.

496

LINEAR MODELS

the parameter spaces 9, 9 0 , and e 1 are as given below, where 0 = (Po, PI' 0'2):

{(Po, Pb ~): -

00 00

9 0 = {(flo, Ph 0'2): 9 1 = 9 - eo.

We must determine A, where

< Po < < Po <

00; - 00 00;

< PI < 00; 0'2 > O} PI = 0; 0'2 > O}

sup L(O; Yl' . , Yn) A = (I eeo sup L(O; Yl' ... , Yn)
(le~

(7)

L(O; Yb ... , Yn) = L(f3o, pb 0'2)

= (2n:2)"12 exp [ -

2~ L (y, - Po - Ptxi}

(I 8)

and the values of Po, PI' 0'2 that maximize this for 0 E 9 are the maximumlikelihood estimates given in Eqs. (7) to (9). Thus we get

-2 = 1 '" 1J - PIX,). 1J 2 where 0' '--' (y, - Po To find sup L(O; Yb ... , Yn), we substitute n (I eeo PI = 0 into Eq. (18) above and get

L(flo, 0'2 ) = (2n0'2t12 exp - 20'2 '--' (y i-PO)2] .

But this is the likelihood function for a random sample of size n from a normal distribution with mean Po and variance ~. The values of Po and (12 that maximize the likelihood function are the maximum-likelihood estimates
p~

[1 '"

= 57

and

Thus sup L(O; Yl, .. " Yn) = (0'*22n)-nI2 exp [(I

e~o

~ L (Yt - p~i] 2(1

= (0'*2 2n) -nI2e-nI2.

TESTS OF HYPOTHESES-CASE A

497

We obtain
A -

&2 )n12 ,- ( 0'*

2

for the generalized likelihood-ratio. Instead of A we will examine the quantity (n - 2)(A -21n - 1), which is a monotonic function of A and hence will give an equivalent test function. We get
L(Yi - y)2 - L(Yi -

L (Yi Replace

Po - Pt XJ2 Po - Pt X i)2
x)F.

Po with Po = y - PtX in the numerator, and get A-21n _ 1 = L (Yi - y)2 - L [(Yi - y) - Pt(x i L (Yi - Po - Pt X j)2
( _ 2)(A -21n _ 1) = PI n

Hence,

L (Xi fl2

X)2 =

PI L (Xi -

2 x)2/O' &2/0'2'

which is the ratio of the values of two independent chi-square random variables (under .J'f0: Pt = 0) divided by their respective degrees of freedom, which are 1 for the numerator and n - 2 for the denominator. Thus en - 2)(A -21n - 1) has an F distribution with 1 and n - 2 degrees of freedom under .J'f 0 . The generalized likelihood-ratio test says to reject .J'f 0 if and only if A < AO, or if and only if en - 2)(A- 2In - 1) > en - 2)(Ao2In - 1) = A~ (say), or if and only if

[PI L (Xi -

x)2]/fl2 > A~,

where A~ is chosen for a desirable size of Type I error. Note that (n - 2)(A -21n - 1) is the square of

Jvar [fiIl'
and recall that the square of a Student's t-distributed random variable with n - 2 degrees of freedom has an F distribution with 1 and n - 2 degrees of freedom. Thus we have verified that if the confidence-interval statement in Eq. (16) is used to test .J'f0: Pt = 0 versus .J'ft : Pt =P 0, it is a generalized likelihood-ratio test. We will generalize this result slightly in the following theorem.

498

UNEAR MODELS

Theorem 3 In the linear model given in Definition 1 the generalized likelihood-ratio test of size ex of .1t0: PI = b 1 (b 1 is a given constant) versus .1t 1: P1 i= bi is given by the following: Use Eq. (16) to set a 1 - ex confidence interval on Pb and reject .1t0 if and only if the confidence interval IIII does not include b t We shall state a theorem concerning a test of hypothesis on proof will be asked for in Prob. 18.

Po,

and the

In the linear model given in Definition 1 the generalized likelihood-ratio test of size ex of .1t0: Po = b o(b ois a given constant) versus .1t1 : Poi=b o is given by the following: Use Eq. (15) to set a I-ex confidence interval on Po, and reject .1t 0 if and only if the confidence interval does not include boo IIII There are many other tests that are of interest for the linear model and the interested reader can consult Refs. 17, 29, 31, and 32.

Theorem 4

POINT ESTIMATION-CASE B

For this case Y1 , Y2 , . , Y n are pairwise uncorrelated random variables with means Po + PI X t, Po + PI X 2, ... , Po + PtXn and variances a2. Since the joint density of the Yj is not specified, maximum-likelihood estimators of Po, Pb and (J'2 cannot be obtained. In models when the joint density of the observable random variables is not given, a method of estimation called least-squares can be utilized.

Definition 2 Least-squares Let (Y" Xi), i = 1, 2, ... , n, be n pairs of

observations that satisfy the linear model given in Definition 1. values of Po and PI that minimize the sum of squares The

are defined to be the least-squares estimators of Po and To find the least-squares estimators of Po and that minimize
1'1

Pl'

IIII

PI' We must find the values

Lcpo, PI) = L (Yi - Po - PI Xt)2,

I"" 1

POINT ESTIMATION-CASE B

499

and clearly these are the same values that maximize the likelihood function in Eq. (5). Hence We have the following theorem. Theorem 5 In Case B of the simple linear model given in Definition 1 the least-squares estimators of Po and PI are given by fio and fib where

fi 1-

L (Yi ~

Y)(Xj - x)
X

L. Xi -

-)2

'

(19)

1111
The least-squares method gives no estimator for (J2, but an estimator of based on the least-squares estimators of Po and P1 is

(J2

For Case A the maximum-likelihood estimators of Po, P1' and (J2 had some desirable optimum properties. The first corollary of Theorem 2 states that fio and fi 1are uniformly minimum-variance unbiased estimators. That is, in the class of all unbiased estimators of Po and Pb the estimators fio and fi1 in Eq. (13) have uniformly minimum variance. No such desirable property as this is enjoyed by least-squares estimators for Case B. For Case A the assumptions are much stronger than for Case B, where the distribution of the random variables Yj is assumed to be unknown; so We should not expect as strong an optimality in the estimators for Case B. For Case B, we shall restrict our class of estimating functions and determine if the least-squares estimators have any optimal properties in the restricted class. Since C[Yd = Po + P1X" We see that Po (and P1) can be given by the expected value of linear functions of the Y i Within this class of linear functions We will define minimum-variance unbiased estimators. Definition 3 Best linear unbiased estimators Let Y1 , Y2 , , Y n be observable random variables such that C[Yj] = rlO), where r;(') are known functions that contain unknown parameters 0 (0 may be vectorvalued). To estimate any OJ in 0, consider only the class of estimators that are linear functions of the random variables Yj In this class consider only the subclass of estimators that are unbiased for OJ. If in this restricted class an estimator of OJ exists which has smaller variance than any other estimator of OJ in this restricted class, it is defined to be the best linear unbiased estimator of OJ ("best" refers to minimum variance). 1111

500

UNEAR MODELS

It should be noted that there are two restrictions on the estimating func-

tions before the property of minimum variance is considered. First, the class of estimating functions is restricted to linear functions of the Yi . Second, in the class of linear functions of the Yi only unbiased estimators are considered. Finally, then, consideration is given to finding a minimum-variance estimator in the class of estimating functions that are linear and unbiased. We will now prove an important theorem that gives optimum properties for the point estimators of /30 and /31 derived by the method of least squares for Case B. This theorem is often referred to as the Gauss-Markov theorem.

Theorem 6 Consider the linear model given in Definition 1, and let the assumptions for Case B hold. Then the least-squares estimators for /31 and /30 given in Eq. (19) are the respective best linear unbiased estimators for /31 and /30 .
We shall demonstrate the proof for /30; the proof for /31 is similar. Since We are restricting the class of estimators to be linear, We haVe :9 0 = I aj Yj We must determine the constant aj such that:
PROOF

(i) 8[:9 0] = /30; that is, :9 0 is an unbiased estimator of /30 . (ii) var [:9 0] is a minimum among all estimators satisfying (i). For (i) We must haVe

This gives the two equations which must be satisfied

and

(20)

Now var [:9 0] = 8[(:9 0 - /30)2] = 8[(I aj Y j

-

/30)2]
-

= 8[[I aj(/3o + /31Xj + E j ) - /3oY] = 8[(/30 I aj + /31 I ajx] + I ajEj

By the restrictions of Eq. (20) var[:9 o] = 8[(I ajE])2]

/30)2J.

= 8[I aJ EJ +
]

I
]

Ii

ajajEjE i ].

j*i

POINT ESTIMATION-CASE B

501

The quantity 8[E jE J ] is 0 if i t= j since, by assumption, the E, are uncorrelated and have means O. Hence var [fJo] =
(12

L aJ.

Since (12 is a constant, to minimize var [fJo] We need to minimize L Thus constants aj must be found which minimize L subject to the restrictions of Eq. (20). Using the theory of Lagrange multipliers, We must . . . nnmnnze

a;

aJ.

=L

aJ - Al (L aJ

1) - A2 L aJ x j '

Taking derivatives, one finds

t = 1,2, ... , n
-

(21)

8L OAI

= -

L aJ + 1 =

0,

8L OA2 = -LajXj=O.
If we sum Over the first n equations, We get (using

L at = 1)
(22)

2=nAl+A2Lxj,
If we multiply the jth equation in (21) by x j and add, We get

2 L x j aJ = Al L x j + A2 L xJ,
or since I ajXj = 0, this becomes

If we substitute this into (22), We get

A _ - 2 I xtln _ - 2x 2 - '\' 2 -2 - '\' ( 2 '-' Xi - nx '-' Xi - Xl

and
1 _

2
'\'

11.1 -

'-' (XI - x)

L xf/n 2

Substituting Al and A2 into the tth equation in (21) and solving for at gives

SOl

LINEAR MODELS

The best linear unbiased estimator of Po is therefore

i'\ _"

DO -

at

y. _

t -"

YL

x; - x L Y
_ 2

L(Xi - x)

t Xt _ -

Y -

i'\-

.01 X,

which is the one given by least squares, and so the proof is complete. similar proof holds for PI .

1111

PROBLEMS
1 Assume that the data below satisfy the simple linear model given in Definition 1 for Case A.
y:

x:
2 3 4 5 6 7
8

-6.1 -2.0

-0.5 0.6

7.2 1.4

6.9 1.3

-0.2 0.0

-2.1 -1.6

-3.9 -1.7

3.8 0.7

9
10

Find the maximum-likelihood estimates of /30, /31, and a 2 , In Prob. 1 find the UMVUE of /30 + 3/31' In Prob. I find a 95 percent confidence interval on /30; on /31; on a 2 In Prob. I find a 90 percent confidence interval on p.(x) for x = -1.0. In the simple linear model for Case A find the maximum-likelihood estimator of 8, where () = /30 + 3/31 + 2a 2 In Prob. 5 find the UMVUE of (). In the simple linear model for Case A, show that p proportion of the distribution of Yat x = Xo is below gp, where gp /30 + /31XO + Zp a and Zp is given by <I>(zp) = p. In Prob. 7 find the UMVUE of gp Use the data in Prob. I to evaluate the UMVUE of gl' in Prob. 7. The hardness Y of the shells of eggs laid by a certain breed of chickens was assumed to be roughly linearly related to the amount x of a certain food supplement put into the diet of the chickens. The model was assumed to be a simple linear model for Case A. Data were collected and are given below:

y,: .70 .98 1.16 1.75 .76 .82 .95 1.24 1.75 1.95
Xi:

.12 .21

.34

.61

.13 .17 .21

.34

.62

.71

11 12 13 14 15

Test the hypothesis that /31 1.00 versus the hypothesis /31 :t 1.00. Use a Type I error probability of 5 percent. In Prob. 10 test the hypothesis /31 > I versus the hypothesis /31 < I. In Prob. 10 test the hypothesis p.(.50) > 1.5 versus the hypothesis p.(.50) 1.5. Use a Type I error probability of 10 percent. In Prob. 10 compute a 90 percent confidence interval on 2a. In the simple linear model for Case A find the UMVUE of /3da 2 Consider the simple linear model given in Definition I except var [Y,] = a/ 1 a l , where at, i = I, 2, ... ., n., are known positive numbers. Find the maximumlikelihood estimators of /30 and /31'

PROBLEMS

503

16 What are the conditions on the Xi in the simple linear model for Case A so that .fio and .fi 1 are independent? 17 In the simple linear model for Case A show that Yand .fit are uncorrelated. Are they independent? 18 Prove Theorem 4. 19 In Theorem 6 give the proof for the best linear unbiased estimator of f31. 20 For the simple linear model for Case B prove that the best (minimum-variance) linear unbiased estimator of f30 + f31 is 130 + 131, where .fio and .fit are the leastsquares estimators of fJo and fJh respectively. 21 Extend Prob. 20 to cofJo + CtfJh where Co and Cl are given constants.

XI
NONPARAMETRIC METHODS

1 INTRODUCTION AND SUMMARY

The important place ascribed to the normal distribution in statistical theory is well justified on the basis of the central-limit theorem. However, often it is not known whether the basic distribution is such that the central-limit theorem applies or whether the approximation to the normal distribution is good enough that the resulting confidence intervals and tests of hypotheses based on normal theory are as accurate as desired. For example, if a random sample of size n is taken from a population with a normal density and a .95 confidence interval is set about the mean (see Sec. 3.1 of Chap. VIII) then the frequency interpretation is the following: If repeated random samples are taken from this population and if a 95 percent confidence interval is obtained for each random sample, in the long run 95 percent of these intervals will contain the mean of the density. If sampling is from a density that is not normal, then, instead of 95 percent of the intervals containing the mean, it may be 99 or 90 percent, or some other percentage. If it is close to 95 percent, say 93 to 97 percent, usually the experimenter will be satisfied. However, if it deviates a large amount from the desired

INTRODUCTION AND SUMMARY

50S

percentage, then the experimenter will probably not be satisfied. In cases where it is known that the conventional methods based on the assumption of a normal density are not applicable, an alternative method is desired. If the basic distribution is known (but is not necessarily normal), one may be able to derive exact (or sufficiently accurate) tests of hypotheses and confidence intervals based on that distribution. In many cases an experimenter does not know the form of the basic distribution and needs statistical techniques which are applicable regardless of the form of the density. These techniques are called nonparametric or distribution-free methods. The term "nonparametric" arises from considerations of testing hypotheSeS (Chap. IX). In forming the generalized likelihood-ratio, for example, one deals with a parameter space which defines a family of distributions as the parameters in the functional form of the distribution vary over the parameter space. The methods to be developed in this chapter make no use of functional forms or parameters of such forms. They apply to very wide families of distributions rather than only to families specified by a particular functional form. The term " distribution-free" is also often used to indicate similarly that the methods do not depend on the functional form of distribution functions. The nonparametric methods that will be considered will, for the most part, be based on the order statistics. Also, although the methods to be presented are applicable to both continuous and discrete random variables, We shall direct our attention almost entirely to the continuous case. Section 2 will be devoted to considerations of statistical inferences that concern the cumulative distribution function of the population to be sampled. The sample cumulative distribution function will be used in three types of inference, namely, point estimation, interval estimation, and testing. Population quantiles have been defined for any distribution function regardless of the form of that distribution. Section 3 deals with distribution-free statistical methods of making inferences regarding population quantiles. Section 4 studies an important concept, that of tolerance limits. The similarities and differences of tolerance limits and confidence limits are noted. In Sec. 5 We return to an important problem in the application of the theory of statistics. It is the problem of testing the homogeneity of two populations. This problem was first mentioned in Subsec. 4.3 of Chap. IX when we tested the equality of the means of two normal populations. I t was considered again in Subsec. 5.3 of Chap. IX when We tested the equality of two multinomial populations. We indicated there that the derived test using a chi-squaretype statistic couId be used to test the equality of two arbitrary populations, and so we had really anticipated this chapter inasmuch as We derived a distributionfree test. Other distribution-free tests of the homogeneity of two populations

506

NONPARAMETRlC METHODS

xi

will be presented in Sec. 5. Included will be the sign test, the run test, the median test, and the rank-sum test. In this chapter we present only a very brief introduction to nonparametric statistical methods. This chapter is similar to the last inasmuch as it includes use of the three basic kinds of inference that Were the focus of our attention in Chaps. VII to IX. We shall see that much of the required distributional theory is elementary, seldom using anything more complicated than the basic principles of probability that Were considered in Chap. I and the binomial distribution.

2 INFERENCES CONCERNING A CUMULATIVE DISTRIBUTION FUNCTION 2.1 Sample or Empirical Cumulative Distribution Function

In Subsec. 5.4 of Chap. VI, We defined the sample cumulative distribution function (c.d.f.). We indicated there that it could be used to estimate thecumulative distribution function from which We sampled. In this subsection some results about the sample c.d.f. will be reviewed and used to formulate point estimates. In the two following subsections the sample c.d.f. will be utilized to test a hypothesis (in Subsec. 2.2) and to set a confidence interval (in Subsec. 2.3). Recall that (see Definition 13 in Chap. VI) the sample c.d.f. is defined by

Fn(x) = - (number of Xi less than or equal to x)

where Xl, ... , Xn is a random sample from some c.d.f. F(). Theorem 17 of Chap. VI,
P [F.(X)

According to

=~] =(;) [F(x)]"[l -

F(x)r"

k = 0, 1, ... , n,

(2)

where Fn( . ) is the sample c.d.f. corresponding to c.d.f. F( .). we see that
S[Fn(x)]

From Eq. (2),

k=O

~ (n)
n k

[F(x)t[1 - F(x)]n-k

F(x)

(3)

and similarly
1 var [Fn(x)] = - F(x)[1 - F(x)]. n
(4)

INFERENCES CONCERNING A CUMULATIVE DISTRIBUTION FUNCTION

507

In fact, since Fn(x) is the sample mean of random variables I(-ro,x](X1), ... ,I( ro, x]( Xn), we know by the central-limit theorem that F n(x) is asymptotically normally distributed with mean F(x) and variance (l/n)F(x)[1 - F(x)]. Equations (3) and (4) show that for fixed x, Fn(x) is an unbiased and mean-squared-error consistent estimator of F(x), regardless of the form of F( . ). If one is interested in estimating F(x) for every x (rather than for a fixed x), then one is interested in saying something about how close Fn(x) is to F(x) jointly Over all values x; hence the following result is of interest:
P[

sup
-ro<x<ro

IFn(x) -

F(x)l

-----+~

0] = 1.

(5)

Equation (5), known as the Glivenko-Cantelli theorem, states that with probability one the convergence of Fn(x) to F(x) is uniform in x. We can define
Dn =

sup
-ro<x<ro

IFix) -

F(x)l.

(6)

Dn is a random quantity that measures how far F n(') deviates from F(')' Equation (5) states that P[lim Dn = 0] = 1; so, in particular, the c.d.f. of Dn ,

say FDn( ), converges to the discrete c.d.f. that has all its mass at O.

In the next

subsection we will consider the limiting distribution of J~ Dn. Equation (5) tells us that the estimating function Fn(x) of the c.d.f. F(x) converges to F(x) uniformly for all x with probability one. Instead of a point estimate of F(x) = P[X ~ x], one might be interested in a point estimate of F(y) - F(x) = P[x < X ~ y] for fixed x < y. The following remark is useful in showing that Fn(Y) - Fix) is an unbiased mean-squarederror consistent estimator of F(y) - F(x).

Remark
cov [Fn(x), Fn(Y)]
PROOF

.1
= - F(x)[1 - F(y)]

for y > x.

(7)

508

NONPAR.AMETRIC METHODS

1 ;;{8[I(-oo,x](X 1)I( oo,yiXl)]

- 8[I(-oo,X](X 1)]8[I( oo,yiXl)]}

= - [F(x) - F(x)F(y)]

=-

1 F(x)[l -.- F(y)]. n

1111

Using Eq. (7), one sees immediately that var [FiY) - Fn(x)] = var [Fn(Y)] - 2 cov [Fix), Fn(Y)]

+ var [Fn(x)]

=-

1 [F(y) - F(x)][l - F(y) n

+ F(x)];

mean-sQuared-error consistency of Fn(Y) - Fn(x) as an estimator of F(y) - F(x) follows immediately. Rather than estimating P[x < X <y], i.e., the probability that X falls in some interval, one might consider estimating P[X E B], i.e., the probability that X falls in some set B.

It can be shown that (seethe Problems) (lfn)

1= 1

L IB(Xi )

is an unbiased estimator of P[X E B], and var [ 1 n ] 1 IB(Xj) = - P[X E B](l - P[X E BD, n i= 1 n

hence, is mean-sQuared-error consistent.

2.. 2 Kolmogorov-Smirnov Goodness-of-fit Test

We noted above that Fn(x) has an asymptotic normal distribution. Equivalently, ~ [Fn(x) - F(x)] has a limiting normal distribution with mean 0 and variance F(x)[l - F(x)]. We now state (without proof) a result that gives the limiting distribution of

In Dn = J~ -oo<x<oo sup IFix) -

F(x)l.

Theorem 1 Let Xh , Xn , be independent identically distributed random variables having common continuous c.d.f. F x( . ) = F( . ). Define
Dn = diXb
... ,

Xn) =

sup I Fn(x) - F(x) I, -oo<x<oo

INFERENCES CONCERNING A CUMULATIVE DISTRIBUTION FUNCTION

509

where F n(x) is the sample c.d.f.

n-+oo

Then
(8)

limF",nf)n(x) = linlP[J~Dn ~ x]
n-+oo

say.

IIII

The c.d.f. given in Eq. (8) does not depend on the c.d.f. from which the sample was drawn (other than that it be continuous); that is, the limiting distribution of J~ Dn is distribution-free. This fact allows Dn to be broadly used as a test statistic for goodness of fit. For instance, suppose one wishes to test that the distribution that is being sampled from is some specified continuous distribution; that is, test .1l' 0: Xi '" F o( .), where F o( .) is some completely specified continuous c.d.f. If.1l' 0 is true,
Kn
=

In(XI ,

. ,

Xn)

In

sup
-oo<x<oo

1Fn(x)

- F o(x) 1

(9)

is approximately distributed as H( . ), the c.d.f. given in Eq. (8). If.1l' 0 is false, then Fn( . ) will tend to be near the true c.dJ. F(') and not near Fo( . ), and consequently sup 1F n(x) - F o(x) I will tend to be large; hence a reasonable
-oo<x<oo

test criterion is to reject .1l' 0 if

Kn
=

sup
-oo<x<oo

IFn(x) - Fo(x) I is large. Since

J~

sup
-oo<x<oo

Fn(x) - F o(x) I is approximately distributed as H(') when

.1l' 0 is true and H(') has been tabulated, k l - ex can be determined so that 1 - H(k l - ex ) = Ct, and hence P[Kn > k l - ex ] ~ Ct. That is, the test defined by "Reject .1l' 0 if and only if Kn > k l - ex " has approximate size Ct. Such a test is often labeled the Kolmogorov-Smirnov goodness-of-fit test. It tests how well a given set of observations fits some specified c.dJ. F o( '). The fit is measured by the so-called Kolmogorov statistic sup IFix) - F o(x)l. Theorem 1 gives
-oo<x<oo

an asymptotic distribution for Dn. tabled for various n. See Ref. 44.

The exact distribution of Dn has been

EXAMPLE 1 A question of at least curious interest is the following: Are the times of birth uniformly distributed over the hours of the day? For 37 consecutive births (actual data) the following times Were observed: 7:02 P.M., 11 :08 P.M., 3:56 A.M., 8:12 A.M., 8:40 A.M., 12:25 P.M., 1 :24 A.M., 8 :25 A.M., 2 :02 P.M., 11 :46 P.M., 10 :07 A.M., 1 :53 P.M., 6 ~45 P.M., 9 :06 A.M., 3:57 P.M., 7:40 A.M., 3:02 A.M., 10:45 A.M., 3:06 P.M., 6:26 A.M., 4:44 P.M., 12:26 A.M., 2:17 P.M., 11 :45 P.M., 5:08 A.M., 5:49 A.M., 6:32 A.M., 12:40 P.M., 1 :30 P.M., 12:55 P.M., 3:22 P.M., 4:09 P.M., 7:46 P.M., 2:28 A.M., 10:06 A.M., 11 :19 A.M., 4:31 P.M. Both the hypothesized uniform c.d.f. and the sample c.dJ. are sketched in Fig. 1.

510

NONPARAMETRJC METHODS

XI

t
1 ------

4:44 P.M.
~--~~----~----~----~~----~----~x

8 A.M.

12 noon

o
FIGURE 1

480

720

1440 min.

One can calculate

The critical value for size (J.. = .10 is greater than 1.22; so, according to the Kolmogorov-Smirnov goodness-of-fit test, the data do not indicate that the hypothesis that times of birth are uniformly distributed throughout the hours of the day should be rejected. IIII The Kolmogorov-Smirnov goodness-of-fit test assumed that the null hypothesis was simple; that is, the null hypothesis completely specified (no unknown parameters) the distribution of the population. One might inquire as to whether such a goodness-of-fit testing procedure can be extended to a composite null hypothesis which states that the distribution of the population belongs to some parametric family of distributions, say {F( . ; 8): 8 E a}. For such null hypotheses, sup I Fn(x) - F(x; 8) 1 is no longer a statistic since it depends
x

In sup 1 Fn(x) x

F(x)1 =

J371~~ - }~~ril ~ .85.

on an unknown parameter 8. An obvious way of removing the dependence on 8 is to replace 8 by an estimator, say 0, similar to what was done in the classical chi-square goodness-of-fit test. The test statistic then becomes sup IF n{x)
x

- F{x;

0) I. The distribution of such a test statistic is not known and, in general,

depends on the hypothesized parametric family. Although some studies (often Monte Carlo) have been reported in the literature, much remains to be done before a Kolmogorov-Smirnov goodness-of-fit test for composite hypotheses becomes a practical testing tool.

INFERENCES CONCERNING A CUMULATIVE DISTRIBUTION FUNCTION

511

2.3

Confidence Bands for Cumulative Distribution Function

Theorem 1 can also be used to set confidence bands on the c.d.f. F( .) sampled from. Let ky be defined by H(ky) = y, where H( . ) is the c.d.f. in Eq. (8). A brief table of y and ky is
y
.99
1.63

.95
1.36

.90
1.22

.85
1.14

.80
1.07

It follows that

P[ J~ sup IF,,(x) x

F(x)1

< ky] ~ y,

but P[

In sup IFn(x) x

F(x) I

< ky]

P[ sup IF,,(x) x

F(x) I

< ky/Jn]

P[F.(X) -

j~

< F(x) < F.(x) + j~ for all x],

noting that

k sup IF,,(x) - F(x) I < JY~ x n

if and only if

for all x.

Using the fact that 0 < F(x) :::; 1, We have

that is, the band with lower boundary defined by L(x) = max [0, F,,(x) - ky/ J~] and upper boundary defined by U(x) = min [F,,(x) + ky/ In,, 1] is an approximate 100y percent confidence band for the c.d.f. F( . ), where the meaning of the confidence band is given in Eq. (10).

512

NONPARAMETRIC METHODS

XI

3 INFERENCES CONCERNING QUANTILES

3.1

Point and Interval Estimates of a Quantile

Throughout this section, we will assume that we are sampling from a continuous c.d.f., say F(')' Recall (see Definition 17 of Chap. II) that the qth quantile of c.d.f. F( . ), denoted by eq , is defined by F(e q ) = q for fixed q, 0 < q < 1. In is called the median. We saw in Subsec. 4.6 of Chap. II particular, for q = t, that quantiles can be used to measure location and dispersion of a c.d.f. For instance, e!, (e q + e.- q )/2, etc., are measures of location, and e.9 - e.., .75 - e.2 5' etc., are measures of dispersion. In Subsec. 2.1, we considered estimating F(x) for fixed x; now, We consider estimating eq such that F(e q ) = q for fixed q. We know that if X is a continuous random variable with c.d.f. F( . ), then the random variable F(X) has a uniform distribution (see Theorem 12 in Chap. V) over the interval (0, 1). Hence F(Yj) has the same distribution as the jth order statistic from a uniform distribution, and We know that 8[F(Yj)] = j/(n + 1). (As usual, Y., ... , Yn are the order statistics corresponding to the random sample X., ... , Xn .) Consequently, we might estimate q with Y [lfj/(n + 1) < q < j if q ~ j/(n + 1). (j + 1)/(n + 1), one could estimate q by interpolating between the order statistics Y j and Yj + . ] A confidence-interval estimate of q can be obtained by using two order statistics, the interval between them constituting the confidence interval. We are interested in computing the confidence coefficient for a pair of order statistics.

e!

e e

P[Yj

< eq < Yk ] = P[F(Yj ) 5:: F(e q) = q < F(Yk )] = 1 - P[F(Y) > q] - P[F(Yk ) < q] = P[F(Yj ) < q] - P[F(Yk ) < q].

Recall that
fYj(Y) =

U _ l)~ ~n _ J)! [F(y)]i-'[l

- F(y)]n- if(y);

hence for Z = F( lJ)

dz dy =f(y) ,

and so,
fz(z) = dz;dY frj(Y) =

U _ l)~~n ~ i)! zi-'(\

zr i

for 0 < z < 1.

INFERENCES CONCERNING QUANTILES

513

Thus,
P[F(Yj )

< u] =
= =

fo fz(z) dz
1 ----f B(j, j + 1)
n u .
0

...

1(1 - z)n- j+ 1

dz

IB.,(j, n - j + 1), Hence,

called the incomplete beta function, which is extensively tabulated.

which is the confidence coefficient of the interval (Yj , Yk ). In practice, of course, we are interested in going in the other direction; that is, for fixed I' pickj and k (and consequently order statistics YJ and Yk ) such that
IBq(j,n

'-j+ 1)- IBq(k,n

-k

1)=1',

and then (Yj , Yk ) is a 1001' percent confidence interval for q . Of course, for arbitrary I' there will not exist a j and k so that the confidence coefficient is exactly y. The confidence coefficient can be obtained another way.

But
P[Yj
:::;;

eq] = P[jth order statistic < eq]

= P[j or more observations <

q]

P[exactly i observations < eq]

i=j

,tj(;) [F(e.)]'[l - F(e,)"-' = .t.(~)qi(l _q)n- i;

=
&=

hence,

Note that a table of the binomial distribution can now be used to evaluate the confidence coefficient.

514

NONPARAMETRIC METHODS

XI

EXAMPLE 2 For a sample of size what is the confidence coefficient of the interval (Y2, Y9 ), which is a confidence-interval estimator of the population median? We have
P[Y 2 ~ ~t < Y9 ]
8 =~ i

10,

(n) (1) 2:

8 =~

(10) (1) i 2:

10

= .9784.

IIII

We have presented one way, using order statistics, of obtaining point estimates or confidence-interval estimates for a quantile. Besides being extremely general in that the method requires few assumptions about the form of the distribution function, the method is extraordinarily simple. No complex analysis or distribution theory was needed; the simple binomial distribution provided the necessary equipment to determine the confidence coefficient. The only inconvenience was the paucity of confidence levels that could be attained.

3.2

Tests of Hypotheses Concerning Quantlles

Let X h . . . , Xn denote a random sample from a probability density function, say f( '). Suppose that it is desired to test that the qth quantile of the population sampled from is a specified value, say~. That is, it is desired to test

where f( .) is unspecified (other than being a probability density function). The confidence-interval method of deriving a test (see Subsec. 3.4 of Chap. IX) can be used; for instance, obtain a percent confidence interval for ~q, and accept :leo if and only if the derived confidence interval contains~. Such a test has size 1 An alternative test is the so-called one-sample sign test. It is a very simple test based on the value of a statistic that represents the nurrlber of the n transformed observations that have a positive sign. To illustrate the principle involved in the sign test, consider testing :leo: ~q = ~ versus :Ie1 : ~q =1= ~ for a random sample Xl' ... , Xn from some unspecified probability density function f( '). Let Z denote the number of X/s that exceed~. Equivalently, Z is the number of Xl - ~, ... , Xn - ~ that have a positive sign. If:le 0 is true, Z has a binomial distribution with parameters nand p = 1 - q = If f(x) dx. So if :Ie 0 is true, one would expect Z to be near np, and hence an intuitively appealing test is to accept :leo if and only if Z is near np. Since the distribution of Z is known, one can determine what is meant by " near" by fixing the size of the test.

1001'

1'.

TOLERANCE UMlTS

515

For example, suppose q = t so that q = e t = median; then a possible test of .1l'o:e t =e versus .1l'l:e t =l=e is to accept.1l'o if and only if jZ-npj = 1Z - nl2j < c, where c is a constant determined by
P[ IZ - nl2l < c] = 1 - a,

where a is the desired size of the test.

Now

P[ IZ - n /21 < c] =

n/2 L+c (n) . (l)j(l)n- ; n/2 -c ) 2 2

j

so c can be determined from a binomial table. (For small sample sizes, not many a's are possible, unless randomized tests are used.) The power function of such a test can be readily obtained since the distribution of Z is still binomial even when the null hypothesis is false; Z has the binomial distribution with parameters nand p = P[X > e]. Such a power function could be sketched as a function of p. Note also that the sign test can be used to test one-sided hypotheses. For instance, in testing .1l' 0: q ~ e versus .1l'1: q > e, the sign test says to reject .1l' 0 if and only if Z, defined as above, is large. Again the power function can be easily obtained.

TOLERANCE LIMITS

An automatic machine in a ball-bearing factory is supposed to manufacture bearings .25 inch in diameter. The bearings are regarded as acceptable from an engineering standpoint if the diameter falls between the limits .249 and .251 inch. Production is regularly checked each day by measuring the diameter of a random sample of bearings and computing statistical tolerance limits L1 and L2 from their samples. If Ll is above .249 and L2 is below .251, the production is accepted. How large should the sample be so that one can be assured with 90 percent probability that the statistical tolerance limits will contain at least 80 percent of the population of bearing diameters? There is a simple nonparametric solution to problems of this kind. In more general terms, let f( . ) be a probability density function, and on the basis of a sample of n values it is desirable to determine two numbers, say Ll and L2 , such that at least .80, say, of the area under f( . ) is between Ll and L 2 On the basis of a sample We cannot be certain that .80 of the area under f( . ) is between Ll and L 2 , but we can specify a probability that it is so.

516

NON PARAMETRIC METHODS

XI

L2 = that

In other words, we want to find two functions Ll = 11(Xb . . . , Xn) and 12(X1, ... , Xn) of the random sample Xb ... , Xn such that the probability

is equal to ,}" for specified'}' and 13.

We summarize with a definition.

Definition 1 Tolerance limits Let Xl, ... , Xn be a random sample from

continuous c.d.f. F() having a density function f(). Let Ll 11(X1, ... , Xn) < L2 = 12(X1, ... , Xn) be two statistics which satisfy: (i) The distribution of F(L 2) - F(Ld does not depend on F( . ). (ii) P[F(L 2) - F(Ld > 13] = '}'. Then Ll and L2 will be defined to be 10013 percent distribution-free tolerance // // limits at probability level'}'.
=

Remark Note that the random quantity F(L2) - F(Ld represents the
area under f( . ) between Ll and L2 . / // / For continuous random variables, order statistics Yj and Yk (j < k) form tolerance limits. To obtain the coefficients 13 and'}' in the definition of tolerance limits, We need the distribution of F(L 2) - F(Ll). Recall that
n1

fyjo Yk(Yj, Yk) = (j _ 1)1 (k - I - j)! (n - k)! x [F(y)F-1[F(Yk) - F(Yj)]k-l- j [1 - F(Yk)]n-kf(Yj)f(Yk). Make the transformation Z = F(Yk ) - F(Yj ) and Y = F(Y), find the joint distribution of Yand Z, and then integrate out Y to get the marginal distribution of Z. The following obtains:
I' (

JZ

_ n1 z) - (k _ 1 _ j)!(n _ k

+ j)! Z

k- 1-

j(

1 - z)

n- k +j

I(O,l)(Z),

(12)

which is a beta distribution with parameters k - j and n - k

P[Z < 13] =

+ j + 1. Now
j

f fz(z) dz = IBp(k - j, n - k +
p

+ 1),

the incomplete beta function, which is tabled.

IBp(k - j, n - k

Also, recall that

+ j + 1) =

f. (~)f3i(1 - f3tk-j
I

Thus for any 13, the probability level'}' can be computed.

TOLERANCE LIMITS

517

EXAMPLE 3 For a random sample of size 5, use (Yi, Ys ) as a tolerance interval for 75 percent of the population; that is, p = .75. What is the corresponding probability level y? We seek
y = P[F(Ys) - F(Y1) > .75]
= =

1 - P[F(Ys) - F(Y1 ) < .75]

1-

1=4 I

(~)(.75y(.25)5-i
k j

.3672.

/III

We might note that, in general,

8[Z]

= 8[F(Yk)] - 8[F(Yj )] = n + 1 - n + 1 = n + 1 .
= : =

k-j

(13)

For Example 3 8[Z]

j-.

EXAMPLE 4 Suppose that it is desired to determine how large a sample must be taken so that the probability is .90 that at least 99 percent of a future day's output of bearings will have diameters between the largest and smallest observations in the sample. The quantities are y = .90 and p = .99, and we want to determine n such that
P[F(Yn )
-

F(Y1 ) >

P]

y,
=

where the density of Z k = n. We get

Y = P[Z >

= F(Yn) - F(Y1 )

is given by Eq. (12) for j

1 and

Pl =

f n(n 1

1)2"'-2(1 - z) dz = 1 - npn-l + (n - l)pn.

If we substitute for y and

p, we get the equation .90 = 1 - n(.99)n-l + (n -

1)(.99),

which can be solved to determine n. The solution is n ~ 388.

IIII

There are similarities and differences between tolerance limits and confidence limits. Tolerance limits, like confidence limits, are two statistics, one less than the other, that together form an interval with random end points. The user of either interval is reasonably confident (the degree of confidence being measured by the corresponding confidence level) that the interval ob~ained contains what it is claimed to. This is where the similarity ends. A confidence interval is an interval thought to contain a fixed unknown parameter value. On

518

NONPARAMETRIC METHODS

XI

the other hand, a tolerance interval is an interval thought to contain a prescribed proportion of the values of the random variable under consideration. In other words, a confidence interval is an interval thought to contain an unknown fixed parameter value that characterizes the distribution of population values, whereas a tolerance interval is an interval thought to contain actual population values, and not some characteristic of them.

5 EQUALITY OF TWO DISTRIBUTIONS 5.1 Introduction

In this section various tests of the equality of two populations will be studied. As we mentioned in Sec. 1 above, we first studied the equality of two populations when we tested that the means from two normal populations Were equal in Subsec. 4.3 of Chap. IX. Then again in Subsec. 5.3 of Chap. IX, we gave a test of homogeneity of two populations. A great many nonparametric methods have been developed for testing whether two populations have the same distribution. We shall consider only four of them; a fifth will be briefly mentioned at the end of this subsection. The problem that we propose to consider is the following: Let Xl, ... , Xm denote a random sample of size m from c.d.f. F x( . ) with a corresponding density function fx( .), and let Y1 , , Y,. denote a random sample of size n from c.d.f. F y(') with a corresponding density function fy( '). (Note that We are departing from our usual convention of using Y's to represent the order statistics corresponding to the X's.) Further, assume that the observations from Fx( . ) are independent of the observations from F y( '). Test.1l' 0: F x(z) = F y(z) for all z versus .1l'1: F x(Z) =1= F y(z) for at least one value of z. In Sec. 2 above We pointed out that the sample c.d.f. can be used to estimate the population c.dJ. In the case that .1l' 0 is true, that is, F x(z) = F y(z), We have two independent estimators of the common population c.d.f., one using the sample c.d.f. of the X's and the other using the sample c.d.f. of the Y's. Intuitively, then, one might consider using the closeness of the two sample c.d.f.'s to each other as a test criterion. Although we will not study it, a test, called the two-sample Kolmogorov-Smirnov test, has been devised that USes such a criterion. We will assume throughout that the random variables under consideration are continuous and merely point out at this time that the methods to be presented can be extended to include discrete random variables as well. In our presentation, we will consider testing two-sided hypotheses and will not consider one-sided hypotheses, although the theory works equally well for one-sided hypotheses.

EQUALITY OF TWO DISTRIBUTIONS

519

5.2 Two-sample Sign Test

The first test that we win consider is the two-sample sign test. We shall See that, in a certain sense, this test is a nonparametric analog of the paired t test (see Prob. 17 in Chap. IX). For this test We assume that the sampling situation is such that the X and Yobservations are paired; that is, we observe (Xl' Yi), ... , (Xn' Y One could think that the X observation is "untreated" (a n ) control) and the corresponding Y observation is "treated," the object of the test being to determine if there is a "treatment" effect. We wish to test -*' 0: F x( . ) = F y( .). Assume that (Xb YJ.), ... , (Xn' Yn) is a random sample from some joint distribution Fx, y( " '). Further assume that Fx. y( " .) is such that P[X> Y] = P[X < Y] = i when -*'0 is true. (ReCall that we are assuming continuous random variables, and then such an assumption is satisfied if X and Yare independent.) Consider a test based on the signs of the differences Xi - Yj, i = 1, ... , n. For instance, define

then Zi has a Bernoulli distribution, and consequently Sn =

L Zi has a binomial
i= 1

distribution with parameters nand p = P[Xi > Yi ]. If -*'0 is true, p = i, and &[Sn] = n12. If the alternative hypothesis is two-sided so that p = P[Xi > Yd can be either larger or smaller than i, then a possible test criterion is to accept -*'0 if Sn is close to n12, that is, accept -*'0 if ISn - nl21 :s; k, where k is determined by fixing the size of the test. k is easily determined from a binomial table, and we have a very simple test of the equality of the two populations. One can See that avoidance of the assumption that Xi and Y, are independent is desirable. For example, Xi might represent an observation on the ith entity before some" treatment" and Yi the observation on the same entity after" treatment." In such a case one is not likely to have independence of Xi and Yj since they are observations taken on the same entity, yet one can sometimes test that there is no " treatment" effect by testing that the" before " and" after" popUlations are the same.

5.3 Run Test

As before let Xl, ... , Xm denote a random sample from Fx( . ) and Y I , ... , Y,. a random sample from F y( .). A rather simple test of -*'0: F x(z) = F y(z) for all z is based on runs of values of X and values of Y. To understand the meaning of runs, combine the m x observations with the n y observations and

520

NONPARAMETRIC METHODS

XI

then order (in ascending order of magnitude) the combined sample. example, if m = 4,and n = 5, one might obtain
y

For
(14)

x.

A run is a sequence of letters of the same kind bounded by letters of another kind except for the first and last position. Thus, in Eq. (14) the ordering starts with a run of one y value, then follows a run of two x values, then a run of one y value, and so on; six runs are exhibited in Eq. (14). It is apparent that if the two samples are from the same population, the x's and y's will ordinarily be well mixed, and the total number of runs will be large. If the two popula-. tions are widely separated so that their range of values does not overlap, then the number of runs will be only two, and, in general, differences between the two populations will tend to reduce the number of runs. Thus the two populations may have the same mean or median, but if the x population is concentrated while the y population is dispersed, there will be a tendency to have a long y run on each end of the combined sample, and there will thus be a tendency to reduce the number of runs. A test then is performed by observing the total number of runs, say Z, in the combined sample and rejecting .1P 0 if Z is less than or equal to some specified number Zo. Our task now is to determine the distribution of Z under .1P 0 in order that for a given test size We may specify Zo. If .1P 0 is true, it can be argued that the possible arrangements of the m x values and n y values are equally likely. It is clear that there are exactly

(m ; n)

such arrangements. To find P[Z =

z], it is necessary now to count

all arrangements with exactly z runs. Suppose z is even, say 2k; then there must be k runs of x values and k runs of y values. To get k runs of x values, the m x's must be divided into k groups. We can form these k groups, or runs, by inserting k - 1 dividers into the m - 1 spaces between the m x values with no more than one divider per space. We can place the k - 1 dividers into the m - 1 spaces in
y values in (;

(~=::)

ways.

Similarly, We can construct the k runs of

=: !) ways.

Any particular arrangement of the k runs of x values

can be combined with any arrangement of the k runs of y values; furthermore, the first run in the combined arrangement can be either a run of x values or a run of y values; hence there are a total of 2 (~ exactly z = 2k runs. Hence

=: D (; =: :) arrangements having

EQUALITY OF TWO DISTRIBUTIONS

521

Similarly, for z odd

m- I) (n - I) (m - 1) (n - I) ( P[Z = z] = P[Z = 2k + I] k k- I + k- I k .
=

(m ;n)
0;,

(16)

To test Jf 0 with size of Type I error equal to (as nearly as possible)

one finds the integer Zo so that

L P[Z = %=2

%0

z] =

0;

(17)

and rejects Jf 0 if the observed value of Z dOes not exceed zo. The computation involved in Eq. (17) can become quite tedious unless both m and n are small. Fortunately, the distribution of Z is approximately normal for large samples, and in fact the approximation is usually good enough for practical purposes when both m and n exceed 10. If Jf 0 is true, the mean and variance of Z are
8[Z]

2mn m+n

+I

(18)

and

vu [] Z =

2mn(2mn - m - n) . (m+n)2(m+n-l)

(19)

The asymptotic normal distribution of Z under Jf 0 has mean and variance given in Eqs. (18) and (19). This asymptotic normal distribution can be used to determine the critical value Zo for large samples. The run test is sensitive to both differences in shape and differences in location between two distributions.

5.4

Median Test

Let Xl, ... , Xm be a random sample from Fx( . ) and 11, ... , ~ be a random sample from F y( .). As in the previous subsection, combine the two samples, and order them. Let Zl < Z2 < ... < Zm+n be the combined ordered sample. The median test of Jf o : Fx(u) = Fy(u) for all u consists of finding the median, say Z, of the z values and then counting the number of x values, say mb which exceed z and the number of y values, say nb which exceed z. If Jf 0 is true, ml should be approximately ml2 and nl approximately n12. We can USe either the statistic Ml or the statistic Nl to construct the test. Let us use Ml = number of X's which exceed Z, the median of the combined sample. If m + n is even,

522

NONPARAMETRIC METHODS

XI

there are exactly (m + n)j2 of the observations (combined x's and y's) greater than the median of the combined sample. (Since we have an even number of continuous random variables, no two are equal, and the median is midway between the middle two.) It can be easily argued that

P[M,

mtl ('::)(m + n~2 - m,)

=

( :++n;/2)
for m + n even and .1f 0 true. A similar expression obtains for m Such a distribution can be used to find a constant k such that

+n

odd.

and our test is given by the following: Reject

.1f 0

m if and only if Ml - 2 >k.

Just as in the run test, an asymptotic normal distribution of Ml can be derived, but we will not study it.

5.5 Rank-sum Test

A very interesting nonparametric test for two samples was described by Wilcoxon and studied by Mann and Whitney. Given two random samples Xl' X 2 , , Xm and Yl, ... , Yn from populations with absolutely continuous c.d.f.'s Fx( . ) and Fy( . ), respectively, one arranges the m + n observations in ascending order and then replaces the smallest observation by 1, the next by 2, and so on, the largest being replaced by m + n. These integers are called the ranks of the observations. Let Tx denote the sum of the ranks of the m x values and Ty the sum of the ranks of the n y values.
(m

Note that Tx

+ Ty =

m+n
j=l

Ij =

+n+

l)(m

+ n)j2; so Ty is a linear function of Tx' We could base a test

Tx is linearly related to another

on either statistic Tx or Ty . Let us use Tx. statistic, which we denote by U. Set

U=

j= 1 i=l

II[Y;.oo)(XJ,

(20)

the number of times an X exceeds a Y. For a given set of observations, let rh r2' ... , rm denote the ranks of the x values, and let Xl, ... , x:n denote the

EQUALITY OF TWO DISTRIBUTIONS

523

ordered x values. Clearly Xl exceeds (rl - 1) y-values, X2 exceeds (r2 - 2) yvalues, and so on, and Xm exceeds (rm - m) y-values. Hence

u=
or

1=

I 1 (ri -I) = I

m.

ri -

I' =

tx -

m(m

+ 1)

To find the first two moments of Tx , we find the first two moments of U.
G[U]

= G[II I[fj' 00)(X1)] = II G[I[fj' oo)(Xi )] = II P[Xi > lj] = IIp = mnp,

where
p

= P[Xi >

Yj ]

P[Y

< x I X = x]fx(x) dx = F f(x)fx(x) dx.

1

If .J'f 0 is true,

P=

f Fx(x)fx(X) dx = fo u du = t

SimilarlY, the variance of U can be found. The derivation is somewhat more complicated since one needs the expected value of U 2 From the mean and variance of U, the mean and variance of Tx can be obtained. If.J'f 0 is true, they are given by

and var [ .Lx

rr]

mn(m

+ n + 1)
12

(23)

The exact distribution of Tx turns out to be a very troublesome problem for large m and n. However, Mann and Whitney have calculated the distribution for small m and n, have shown that Tx is approximately normally distributed for large m and n, and have demonstrated that the normal approximation is quite accurate when m and n are larger than 7. Thus for samples of reasonable size one can use the normal approximation with mean and variance given by Eqs. (22) and (23) to find a critical region for testing .J'f 0: F x(z) = F y(z) for all z versus .J'f1: F x(Z) =1= F y(z). The test would be the following: Reject .J'f 0 if I Tx - G[Tx] I is large;

524

NON PARAMETRIC METHODS

XI

that is, Reject :K0 if and only if 1 Tx - 8[Tx] I ~ k, where k is determined by fixing the size of the test and using the asymptotic normal distribution of Tx.

EXAMPLE 5 Find the exact distribution of Tx under :K 0 for m = 3 and n = 2. Each of the following arrangements is equally likely if :K0 is true:

x x x y y, x x y x y, x x y y x, x y x x y, x y x y x, x y y x x, y x x x y, y x x y x, y x y x x, y y x x x.
The corresponding Tx values are, respectively, 6, 7, 8, 8, 9, 10,9, 10, 11, 12; so

P[ Tx = 6] = P[ Tx = 7] =
and

1 1 0'

P[Tx = 8] = P[Tx = 9] = P[Tx = 10] = P[Tx = 11]

=

lo'
IIII

P[Tx = 12] =

/0'

PROBLEMS
1

1 n Show that T= - L IB(X,) is an unbiased estimator of P[X E B]. Find var n ,= 1 and show that T is a mean-squared-error consistent estimator of P[X E B]. 1 n 2 Define Fn(BJ) = - L IBiXt) for j = 1, 2. Find cov [Fn(B 1), Fn(Bz)].
n,=l

rll,

3 Let Y1 , , Yn be the order statistics corresponding to a random sample of size n from a continuous c.d.f. F(' ). (a) Find the density of F( Yj ). (b) Find the joint density of F( Y t ) and F( Y j ). ) - F(Y1 )]. (c) Find the density of [F(Yn) - F(Yz)]/[F(Yn 4 Let Xl, ... , Xn be independent and identically distributed random variables having common continuous c.d.f. F( .). Let YI < ... < Yn be the correSpOnding order statistics, and define F n(') to be the sample c.d.f. Set Dn = sup 1Fn(x) - F(x) I.
-CXl<X<CXl

(a) (b)
(c)

Find the exact distribution of Dn for n = 1. Do the same forn = 2. HINT: Does Dn = max [F(YI ), 1- F(Y1),F(Yz) -I,
1 - F(Yz)]?

Argue that the exact distribution of Dn will not depend on F(')'

PROBLEMS

525

5 Show that the expected value of the larger of a random sample of two observations from a normal population with mean 0 and unit variance is

VV;:;' and hence that

for the general normal population the expected value is fL + a/V;:;'. 6 If (X, Y) is an observation from a bivariate normal population with means 0, unit variances, and correlation p, show that the expected value of the larger of Xand Y 7 is V(l - p)/Tr. We have seen that the sample mean for a distribution with infinite variance (such as the Cauchy distribution) is not necessarily a consistent estimator of the population mean. Is the sample median a consistent estimator of the population median? Construct a (approximate) 90 percent confidence band for the data of Example 1. Does your band include the appropriate uniform distribution? Let Yl < ... < Y s be the order statistics corresponding to a random sample from some continuous c.d.f. Compute P[Yl < fso < Y s] and P[Yz < fso < Y 4 ]. Compute P[Yl < fzo < Y z ]. Compute P[Y3 < f7S < Ys]. Let Yl and Y,. be the first and last order statistics of a random ~~ple of size n from some continuous c.d.f. F(). Find the smallest value o'f n such that P[F( Y,.) -F( Yl ) > .75] > .90. Test as many ways as you know how at the 5 percent level that the following two samples came from the same population:
x
y

8 9

10

11

1.3 1.6

1.4
1.8

1.4
2.0

1.5 2.1

1.7 2.1

1.9

1.9

2.2

2.3

12 Let Xl, ... , Xs denote a random sample of size 5 from the density f(x; fJ) = 1(6 - t. 6 + t)(x). Consider estimating O. (a) Determine the confidence coefficient of the confidence interval (Yl , Ys). (b) Find a confidence interval for 0 that has the same confidence coefficient as in part (a) using the pivotal quantity (Yl + Y s )/2 - O. (c) Compare the expected lengths of the confidence intervals of parts (a) and (b). 13 Find var [U] when F x (') - FA). See Eq. (20). 14 Equation (21) shows that Uand Txare linearly related. Find the exact distribution of U or Tx when JIt' 0 is true for small sample sizes. For example, take m = 1, n = 2; m = 1, n = 3; m = 2, n = 1 ; m = 3, n = 1; and m = n = 2. 15 We saw that G[U] = mnp. Is U/mn an unbiased estimator of p = P[Xt > Y j ] whether or not JIt' 0 is true? Is U a consistent estimator of p? 16 A common measure of association for random variables X and Y is the rank correlation, or Spearman's correlation. The X values are ranked, and the observations are replaced by their ranks; similarly the Yobservations are replaced by their ranks. For example, for a sample of size 5 the observations
x
y

20.4

19.7 8.9

21.8

20.1

20.7

9.2

11.4

9.4

10.3

526

NONPAllAMETRIC METHODS

XI

are replaced by
r(x) r(y)

3 2

1 1

5
5

2 3

4
4

Let r(Xt ) denote the rank of XI and r( Y t ) the rank of Y t Using these paired ranks, the ordinary sample correlation is computed: Spearman's correlation = S = -::::========;;;;:====== ,
[r(XI) - i(X)]Z [r( Yt) - i( Y)]2 [r(XI) - i(X)][r( Yt) - i( Y)]

where f(X) = 2: r(Xt)!n and f(Y) = 2: r(Yt)/n. (a) Show that S = 1 - 6 2: D1/(n 3 - n), where D t = r(Xt) - r(Yt ). (b) Compute the ordinary correlation and Spearman's correlation for the above data. 17 ArgUe that the distribution of S in Prob. 16 is independent of the form of the distributions of X and Y provided that X and Yare continuous and independently distributed random variables. Hence S can be used as a test statistic in a nonparametric test of the null hypothesis of independence. 18 Show that the mean and variance of S (in Prob. 17) under the hypothesis of independence are 0 and 1/(n - 1), respectively.

APPENDIX

MATHEMATICAL ADDENDUM

INTRODUCTION

The purpose of this appendix is to provide the reader with a ready reference to some mathematical results that are used in the book. This appendix is divided into two main sections: The first, Sec. 2 below, gives results that are, for the most part, combinatorial in nature, and the last gives results from calculus. No attempt is made to prove these results, although sometimes a method of proof is indicated.

2 NONCALCULUS

2.1
7

Summation and Product Notation

A sum of terms such as n3 + n4 + ns

~ ni.
1=3

+ n6 + n7

is often designated by the symbol

is the capital Greek letter sigma, and in this connection it is often called the

summation sign. The letter I is called the summation index. The term fol1owing L is called the summand. The" i = 3 " below ~ indicates that the first term of the sum is obtained by putting i = 3 in the summand. The" 7 " above the L indicates that the

528

MATHEMATICAL ADDENDUM

APPENDIX A

final term of the sum is obtained by putting i = 7 in the summand. The other terms of the sum are obtained by giving ithe integral values between the limits 3 and 7. Thus

L: (-I)J-2jx1J = J=2

2x4 - 3x6 + 4x8 - 5x10

An analogous notation for a product is obtained by substituting the capital Greek letter for L:. In this case the terms resulting from substituting the integers for the index are multiplied instead of added. Thus

EXAMPLE 1 Some useful formulas involving summations are listed below. They can be proved using mathematical induction.

i
l.d

i = n(n + 1) .
2

(1)

i
1=1

i2 = n(n + 1)(2n + 1) . 6

(2)

t~1

~ ;4

n(n + 1)(2n + 1)(3n 2 + 3n - 1). 30

(4)

Equation (1) can be used to derive the following formula for an arithmetic series or progression:

" [a + (j L: J=1

l)d] = na

2n(n-l).
1 -r" 1- r

(5)

A companion series, the finite geometric series, or progression, is given by

II-I

L: ar} = a J-O

(6)

II1I
2.2 Factorial and Combinatorial Symbols and Conventions
A product of a positive integer n by all the positive integers smaller than it is usually denoted by n! (read" n/actorial"). Thus
n!

= n(n -

II-I

1)(n - 2) .... 1=

J=O

n (n -

j).

(7)

O! is defined to be 1.

NONCALCULUS

529

A product of a positive integer n by the next k - 1 smaller positive integers is usually denoted by (n)" ' Thus (n)" = n(n 1) ..... (n - k + 1)

" =n

(n

1),

(8)

J=1

Note that there are k terms in the product in Eq. (8). Remark
(n)"

= n!/(n -

k)!, and (n)n = n!/O!

= nJ.

The combinatorial symbol

(;) is defined as follows:

(k

n) = (nh
k!
(n

n! k)!k!'

(9)

(;) is read .. combination of n things taking k at a time" or more briefly as

" n pick k"; it is also called a binomial coefficient.

Define or
k

if k<O

n.

(10)

11/1
Remark

(~) =
(;) =

(:)

1.

(n ~k)'
=

(n : I)

(~) + (k ~ I)

for n

1, 2, ..

and k = 0,

1,

2, ., ..
(11)

Equation (11) is a useful recurrent formula that is easily proved.

/III

Both (n). and the combinatorial symbol (;) can be generalized from a positive integer
n to any real number t by defining
(I)" = t(1 - 1) ..... (I - k

1),

/) (k

t(t - 1) ..... (t - k

+ 1)
(12)

k!

for k = 1, 2, ... , and

(~) = !for k = O.

530

MATHEMATICAL ADDENDUM

APPENDIX A

Remark

- n) = (- n)( - n - 1) ..... ( - n ~ k ( k k!
n(n

+ 1)
1)

= (-1)~ - - - - k - ! - - - -

1) ..... (n

+k -

= (-I)'
2.3 Stirling's Formula

r k-1)
k

II/I

In finding numerical values of probabilities, one is often confronted with the evaluation of long factorial expressions which can be troublesome to compute by direct multiplication. Much labor may be saved by using Stirling's formula, which gives an approximate value of n!. Stirling's formula is

or

where 1 - 1/(12n 1) < r(n) < 1. To indicate the accuracy of Stirling's formula, 10! was evaluated using five-place logarithms and Eq. (13), and 3,599,000 was obtained. The actual value of 10! is 3,628,800. The percent error is less than 1 percent, and the percent error will decrease as n increases.

2.4 The Binomial and Multinomial Theorems

The binomial theorem is often given as
(a

+ b)" = J=O i (~) aJb",-J ]

(15)

for n, a positive integer. The binomial theorem explains why the (;) are sometimes called binomial coefficients. Four special cases are noted in the following remark.

Remark
(1 - t)'" =

(16)

}."O

(~) (-1)JtJ, J

(17)

2'" =

}""O

(~), J

(18)

CALCULUS

531

and

0= i: (-1)1(~).
}=o

(19)

1111
Expanding both sides of
(l

+ x)"(l + xt = (l + X)Hb

and then equating coefficients of x to the nth power gives

}=o

i (a) (nbj ) = (a + b) j n'

(20)

a formula that is particularly useful in considerations of the hypergeometric distribution. A generalization of the binomial theorem is the multinomial theorem, which is

")" n'" a} =.2,,' ( .2 IT n a;',

}=1

(21)

l= 1

nd

1",1

where the summation is over all nonnegative integers nl, nz, ... , n" which sum to n. A special case is

Also note that

3 CALCULUS 3.1 Preliminaries

It is assumed that the reader is familiar with the concepts of limits, continuity, differenti~ ation, integration, and infinite series. A particular limit that is referred to several times in the book is the limit expression for the number e; that is,
lim (l
X"" 0

X)l/X

= e.

(24)

Equation (24) can be derived by taking logarithms and utilizing l'Hospital's rule,

532

MATHEMATICAL ADDENDUM

APPENDIX A

which is reviewed below. There are a number of variations of Eq. (24)t for instance, lim (1
x-+ a:J

+x

1)~

=e

(25)

and lim (1
X'" 0

+ Ax)1/x =

e).

for constant A.

(26)

A rule that is often useful in finding limits is the following so-called I'Hospital's rule: If f( ) and g( . ) are functions for which lim f(x) = lim g(x) = 0 and if
' f'(x) IIm-x-+a g'(x)

exists, then so does

, f(x) II m x ... a g(x) ,

and
lim f(x) x-+a g(x)

lim f'(x) x'" a g'(x) .

EXAMPLE 2 Find lim [(llx)

"" ... 0

lo~ (1

x)].

Let f(x) =

lo~ (l

x) and g(x) = x;

then
, f'(x) IIm-x'" 0 g'(x)
Let
1(1)

lim
x-+ 0

_1_ = I = lim [~IO~ (l + X)] . 1+x

x'" 0

IIII

Another rule that we use in the book is Leibniz' rule for differentiating an integral:

lI(t)

f(x; t) dx,

(I(t)

where f( ; . ), g( . ), and h( ) are assumed differentiable.

Then
dg f(g(t);t) dt .
(27)

- =
dt

dl

JlI(t) (I(t)

of

dh dx+ f(h(t); t)ot dt

Several important special cases derive from Leibniz' rule; for example, if the integrandf(x; t) does not depend on t, then
d [ dt

{(t)

h(t)

f(x) dx

dh f(h(t dt

f(o(t)) dt ;

dg

(28)

in particular, if g(l) is constant and h(t) = t, Eq. (28) simplifies to

[f

[(x)

dx] = [(/),

(29)

CALCULUS

533

3.2

Taylor Series

The Taylor series for [(x) about x = a is defined as

[(x)

= [(a) + [(1)(a)(x -

a) +

[(2)(a)(x - a)2

2!

[(")(a)(x - a)"

---,--+R", n.

(30)

where and
a<c<x.

R" is called the remainder. f(x) is assumed to have derivatives of at least order n + l. If the remainder is not too large, Eq. (30) gives a polynomial (of degree n) approxima tion, when R" is dropped, of the function [(.). The infinite series corresponding to Eq. (30) will converge in some interval if lim Rn 0 in this interval. Several important
w

,,-+

<Xl

infinite Taylor series, along with their intervals of convergence, are given in the following examples.

EXAMPLE 3 Suppose [(x) = eX and a

O.

Then

~=

X2 1 +x+-

x"
31
00

21

=2j!
)=0

<Xl

x)
for -

<x<

00.

(31)

1II1
EXAMPLE 4 Suppose [(x) = (1- x)t and a = 0; then [(1)(x) = t(1 x)t-l, [(2)(X) = t(t l)(1 X)t-2, .. , [U)(x) =( -l)Jt(t - l)'" (t - j + 1)(1 - x)t- J, and hence
[(x)

(l-xr=

)=0

<Xl

x) (-1)1(1)-.

J!

)=0

i (t.)(_X)J
]

for -1 <x < 1.

(32)

I111
There are several interesting special cases of Eq. (32).
t = -

n gives
(33)

i
t

i-I)

xJ

for -1 <x < 1;

= - 1 gives the geometric series

(1 - X)-l =

L
):;0

<Xl

xJ;

(34)

534

MATHEMATICAL ADDENDUM

APPENDlX A

-2 gives
l

(1 - X)-2

~
J-O

U + l)xJ

(35)

EXAMPLE 5 Suppose f(x)

lo~ (1

l0ie (1

x) and a

= 0; then
for -1

+ x) = x - "2

x2

<x< 1.

(36)

II/!
The Tay lor series for functions of one variable given in Eq. (30) can be generalized to the Taylor series for functions of several variables. For example, the Taylor series for f(x, y) about x = a and y = b can be written as
f(x,y) f(a, b)

+ f;t;(a, b)(x -

a)

~(a, b)(y - b)

+
+ f.".,,(a, b)(y b)2]

2! f;t;;t;(a, b)(x - a)2 + 2fx,,(a, b)(x

where

a)(y - b)

+ ... ,

and similarly for the others.

3.3 The Gamma and Beta Functions

The gamma function, denoted by r( . ), is defined by
r(t)
=

J xt -Ie-;t; dx
l

for t

> O.

(37)

r(t) is nothing more than a notation for the definite integral that appears on the right-

hand side of Eq. (37).

Integration by parts yields

r(t + 1) = tr(t),
and, hence, if t

(38)

= n (an integer),
r(n

+ 1) = n!.
1) ~
I1T,

(39)

If n is an integer,
1 . 3 . 5 ..... (2n
r(n +!)

211

(40)

CALCULUS

535

and, in particular,

The beta function, denoted by B( ., .), is defined by

B(a, b) =

10

xa-1(l -

X)b-l

dx

for a > 0, b > O.

(42)

Again, B(a, b) is just a notation for the definite integral that appears on the right-hand side of Eq. (42). A simple variable substitution gives B(a, b) = B(b, a). The beta function is related to the gamma function according to the following formula =
r(a)r(b) B(a, b) = r(a + b) (43)

APPENDIX

TABULAR SUMMARY OF PARAMETRIC FAMILIES OF DISTRIBUTIONS

1 INTRODUCTION
The purpose of this appendix is to provide the reader with a convenient reference to the parametric families of distributions that were introduced in Chap. III. Given are two tables, one for discrete distributions and the other for continuous distribu~ tions.

538

PARAMETRIC FAMILIES OF DISTRIBUTIONS

APPENDIX B

Table 1 DISCRETE DISTRIBUTIONS Name of parametric family of distributions Discrete uniform

Discrete density functions f()

f(x)

Parameter Space
N= 1,2, ...

Mean fL = ~[X]
N+1

=N

l{l .... N}(X)

Bernoulli

f(x)

= pXql-xl{o.

l}(x)

O<p<l (q = 1- p) O<p< 1 n = 1,2,3, .,. (q = 1 - p)

Binomia1

l(x)

=(~)PXq"-XI{o. 1 . n,{x)

np

Hypergeometric

(~)e\!=:)
flx) =

(~)

l{o.

1 .....

n}(X)

M=1,2, ... K = 0, 1, ... , M n=1,2, ... ,M

K nM

Poisson

e-AAx f(x) =-,-l{o. 1 )(x) x.

A>O

Geometric

f(x)

= pl(o. 1. ... }(x)

O<p<l (q = 1- p)

q p

Negative binomial

f() _(r+x-1) x x p '/(0.1 ) (x)

O<p<l r> 0 (q = 1 - p)

rq p

DISCRETE DISTRIBUTIONS

539

a 2 =tf[(X - fL)2]

Variance

Moments fL: = tf[xr] or fLr and/or cumu]ants Kr

,
fL3

= tf[(X - fLY]

Moment generating function

tf[e fX ]

N2-1

12

N(N + 1)2 4
(N

+ 1)(2N + 1)(3N + 3N 2

1)

30
pq fL: = p

for an r

npq

fL3 = npq(q - p) fL4 = 3n 2p2q2 npq(1- 6pq)

K M-KM-n nM M M-1

"[X(X - I)

o(X - r

+ I)J

~ r! (~y)

not useful

Kr=A fL3 =A fL4 = A q p2

for r

1,2, '"

exp[A(e t -l)]

+ 3A2
P 1- qe t

q+q2 fL3 =

---pz-

fL4 =
rq p2

q+ 7q2 +q3 p4 r(q

fL3 = r[q

+ q2)
p3

(1
p4

!qetf

fL4 =

+ (3r + 4)q2 + q3]

S40

PARAMETRIC FAMILIES OF DISTRIBUTIONS

APPENDIX B

Table 2 CONTINUOUS DISTRIBUTIONS Name of parametric family of distributions Uniform or rectangular

Cumulative distribution function F(o) or probability density functionf(o)

f(x)

Parameter space
-00< a< b

Mean

p, = < 00
2

tB'[X]

= -bl(a,b](x) -a

a+b

Normal
"I~"'" .i
-'
~

f(x)

1
V21TU

exp[ - (x- p,)2/2u 2]

-oo<p,<oo
u>O

p,
1

Exponential

f(x)

= >"e-A:X:I(oo~)(x) = r(r)x"-le-A:X:I(oo~)(x)
=

>">0 >">0
r>O a>O b>O

X
r

Gamma

f(x)

>..r

X
a a+b

Beta

f(x)

1 B(a, b)x"-l(l- x)b-ll(o.l)(X) 1 + [(x -

Cauchy

f(x)

1T~{1

ex)/PJ2}

~>O

-00< ex < 00

Does not exist exp[p.+ lu2]

Lognormal

f(x)

=
1TU
exp[-(1o8eX-P,)2/2u2]I(o.~)(x)

-00 < p,< 00

u>O

1
x

v2

Double exponential

1 (Ix-ex') f(x) =2~ exp ~

-oo<ex<oo
~> 0

ex

CONTINUOUS DIsTRmunONs

541

Variance Moment generating 2 a____rS'_[_(X ____ fL_)2_]____a_n-d-/o-r-c-u-m-u-1-an-t-s-K-r----____________ f_un_ct __ io_n_rS'_[_e_tX_]_________ ~

(b - a)2

Moments fL; = rS'[XT] or fLr = rS'[(X - fLY]

fLr = 0

for r odd
(b - a)t

12

fLr = 2r(r + 1)

(b-aY

for r even
r! U (r/2)! 2r/2' r even;

fLr = 0, r odd; fLr

Kr=

exp[fLt + I

(J2

t 2]

0, r > 2
r(r+ 1)

"2
ab
(a

n',r-

Ar

" for t<" A-t

, r(r + j) fLl = "lr(r)

B(r + a, b)

+ b + 1)(a + b)2

fLr =

B(a, b)

not usefu1 Characteristic function is e'lZt-Pltl not usefu1

Does not exist

Do not exist

exp[2fL + 2a2 ] -exp[2fL +a 2 ]

fLr = 0 for r odd; fLr = r! f3r for r even

e lZt 1 - (f3t)2
( continued)

542

PARAMETRIC FAMIUES OF DISTRIBUTIONS

APPPENDIX B

Table 2 CONTINUOUS DISTRIBUTIONS (continued) Name of parametric family of distributions Weibul1

Cumulative distribution function F(') or probability density functionf()

f(x) = abx"-lexp[-axbJI(o,oo)(x)

Parameter space
a>O b>O

Mean p. = rS'[X]
a- 1IbI'(1

+ b- 1 )

Logistic

F(x)

= [1

e -(x- Cl)/II]-1

-oo<ex<OO

{3> 0
Xo

ex
8xo

Pareto

8x& f(x) = x'+1 1(Jl:o'oo)(x)

> 0 8>0

8-1
for 8> 1

Gumbel or extreme value

F(x) = exp (_e-(JI:-IX)/II) I'[(k + 1)/2] I'(k/2)

-oo<ex<OO

ex
y

+ {3y,
~

{3>0
1
(1

.577216

t distribution

f(x) =

V k1r

+ x 2/k)(k +1)12

k>O

p.=O for k> 1

,

F distribution

f(x) = I'[(m + n)/2] (m)m/2 I'(m/2)I'(n/2) n x(m-Z)/2 x [1 + (m/n)x](m+n)/Z l(o.oo)(x)

m, n =

1. 2, ...

n n-2

for n > 2

~~
7~~

Chi-square distribution

f(x)=

I'(kj2)

(~kIZx"IZ-le-(l/2)XI

(0,00)

(x)

1,2, ...

, 1lI
"

CONTINUOUS DISTRIBUTIONS

543

Variance
a2

= t8'[(X - p.)Z]

Moments p.: = 8[X'] or p., = 8[(X - p.)"] and/or cumuJants K,

Moment generating function &[e tx ]

t8'[Xt] = a-rfbr( 1

a- Z1b [r(1

- rZ(1
fl Z1J'2 3

+ b-

+ 2b1)]

1)

p.: = a-r1br( 1 +~)

+ i)

e llt 1J'fll csc(1J'flt)

8xS
(8 - 1)2(8 - 2)
for 8> 2

for 8> r p.:=O -r

8x'O

does not exist

1J'Z{32 6
k k-2

Kr = ( - fly~(r-l)(l)

for r 2, where ~(.) is digamma function

ellfrO - fl/) for t < 1/{3

p'r

= 0 for k>

r and r odd

kr/ZBr + 1)/2, (k - r)/2) p.r= B(!, k/1J

does not exist

for k > 2
2n Z(m+n 2) m(n - 2)2(n - 4)

for k> rand r even

, =(!:)" r(m/2+ r)r(n/2 p., m r(m/2)r(n/2)

n for r <2 2Jr(k/2 + j) p.) r(k/2)
f _

r)

does not exist

for n > 4

2k

(~r/2
1
21

for t< 1/2

APPENDIX

REFERENCES AND RELATED READING

MATHEMATICS BOOKS
and MORREY: "Calculus with Analytic Geometry: A Second Course," Addison-Wesley Publishing Company, Inc., Reading, Mass., 1971. 2. THOMAS: "Calculus and Analytic Geometry," alternate ed., Addison-Wesley Publishing Company, Inc., Reading, Mass., 1972. 3. WIDDER: "Advanced Calculus," 2d ed., Prentice-Hall, Inc., Englewood Cliffs, N.J., 1961. 4. WYLIE: "Advanced Engineering Mathematics," 3d ed., McGraw-Hill Book Company, New York, 1966.
PR01TER

1.

PROBABILITY BOOKS
5. 6. "Basic Probability Theory," John Wiley & Sons, Inc., New York, 1970. DRAKE: "Fundamentals of Applied Probability Theory," McGraw-Hill Book Company, New York, 1967. 7. DWASS: "Probability: Theory and Applications," W. A. Benjamin, Inc., New York,
ASH:

1970.

PROBABILITY AND

STATISTICS BOOKS 545

8. FELLER: "An Introduction to Probability Theory and Its Applications," Vol. 1, 3d ed., John Wiley & Sons, Inc., New York, 1968. 9. FELLER; "An Introduction to Probability Theory and Its Applications," Vol. 2, John Wiley & Sons, Inc., New York, 1966. 10. PARZEN: "Modern Probability Theory and Its Applications," John Wiley & Sons, Inc., New York, 1960.

PROBABILITY AND STATISTICS BOOKS

Advanced (more advanced than MOB) 11. CRAMER: "Mathematical Methods of Statistics," Princeton University Press, Princeton, N.J., 1946. 12. FERGUSON: "Mathematical Statistics, A Decision Theoretic Approach," Academic Press, Inc., New York, 1967. 13. KENDALL and STUART: "The Advanced Theory of Statistics, Vol. 1, Distribution Theory," 2d ed., Hafner Publishing Company, Inc., New York, 1963. 14. KENDALL and sTUART: "The Advanced Theory of Statistics, Vol. 2, Inference and Relationship," Hafner Publishing Company, Inc., New York, 1961. 15. KENDALL and STUART: "The Advanced Theory of Statistics, Vol. 3, Design and Analysis, and Time Series," Hafner Publishing Company, Inc., New York, 1966. 16. LEHMANN: "Testing Statistical Hypotheses," John Wiley & Sons, Inc., New York, 1959. 17. RAO:" Linear Statistical Inference and Its Applications," John Wiley & Sons, Inc., New York, 1965. 18. WILKS: "Mathematical Statistics," John Wiley & Sons, Inc., New York, 1962. 19. ZACKS:" The Theory of Statistical Inference," John Wiley & Sons, Inc., New York, 1971.

Intermediate (about the same level as MOB) 20. BRUNK: "An Introduction to Mathematical Statistics," 2d ed., Blaisdell Publishing Company, a division of Ginn and Company, Waltham, Mass., 1965. 21. DWASS: "Probability and Statistics," W. A. Benjamin, Inc., New York, 1970. 22. HOGG and CRAIG: " Introduction to Mathematical Statistics," 3d ed., The Macmillan Co. of Canada, Limited, Toronto, 1970. 23. LINDGREN: "Statistical Theory," 2d ed., The Macmillan Company, New York, 1968. 24. TUCKER: "An Introduction to Probability and Mathematical Statistics," Academic Press, Inc., New York, 1962.

546

REFERENCES AND RELATED READING

APPENDIX C

Elementary (less advanced than MOB, but calculus prerequisite)

25. "Mathematical Statistics," 2d ed., Prentice-Hall, Inc., Englewood Cliffs, N.J., 1971. 26. GUlTMAN, WILKS, and HUNTER: " Introductory Engineering Statistics," 2d ed., John Wiley & Sons, Inc., New York, 1971. 27. HOEL:" Introduction to Mathematical Statistics," 3d ed., John Wiley & Sons, Inc., New York, 1971. 28. HOEL, PORT, and STONE: "Introduction to Statistical Theory," Houghton Mifflin Company, Boston, 1971.
FREUND:

SPECIAL BOOKS
29. and WOOD: "Fitting Equations to Data; Computer Analysis of Multifactor Data for Scientists and Engineers," Interscience Publishers, a division of John Wiley & Sons, Inc., New York, 1971. DAVID: "Order Statistics," John Wiley & Sons, Inc., New York, 1970. DRAPER and SMITH: "Applied Regression Analysis," John Wiley & Sons, Inc., New York, 1966. GIBBONS: "Nonparametric Statistical Inference," McGraw..Hill Book Company, New York, 1971. GRAYBILL: "An Introduction to Linear Statistical Models," Vol. 1, McGraw-Hill Book Company, New York, 1961. JOHNSON and KOTZ:" Discrete Distributions," Houghton Mifflin Company, Boston, 1969. JOHNSON and KOTZ: "Continuous Univariate Distributions-I," Houghton Mifflin Company, Boston, 1970. JOHNSON and KOTZ: "Continuous Univariate Distributions-2," Houghton Mifflin Company, Boston, 1970. KEMPTHORNE and FOLKS: U Probability, Statistics, and Data Analysis," The Iowa State University Press, Ames, 1971. .MORRISON: "Multivariate Statistical Methods," McGraw-Hill Book Company, New York, 1967. RAJ: "Sampling Theory," McGraw-Hill Book Company, New York, 1968.
DANIEL

30. 31. 32. 33. 34. 35. 36. 37. 38 . 39.

PAPERS
40.
JOINER

and ROSENBLATT: "Some Properties of the Range in Samples from Tukey's Symmetric Lambda Distributions," Journal 0/ the American Statistical Associa lion, Vol. 66 (1971), 394-399.

BOOKS OF TABLES

547

41.

"The Estimation of the Location and Scale Parameters of a Continuous Population of Any Given Form," Biometrika, Vol. 30 (1939), pp. 391--421. 42. WOLFOWITZ:" The Minimum Distance Method," Annalso/Mathematical Statistics, Vol. 28(1) (1957), pp. 75-88. 43. ZEHNA:" Invariance of Maximum Likelihood Estimation ,"Annals 0/ Mathematical Statistics, Vol. 37 (1966), p. 744.
PITMAN:

BOOKS OF. TABLES

44. "Handbook of Tables for Probability and Statistics," Chemical Rubber Company, Cleveland, 1966. 45. MOLINA: "Poisson's Exponential Binomial Limit," D. Van Nostrand Company, Inc., Princeton, N.J., 1942. 46. OWEN: "Handbook of Statistical Tables," Addison-Wesley Publishing Company, Inc., Reading, Mass., 1962.

APPENDIX TABLES

1 DESCRIPTION OF TABLES Table 1 Ordinates of the Normal Density Function This table gives values of

for values of x between 0 and 4 at intervals of .01. For negative values of x one uses the fact that r$( - x) = r$(x). Table 2 Cumulative Normal Distribution This table gives values of

for values of x beteenn 0 and 3.5 at intervals of .01.

For negative values of x, one uses

DESCRIPTION OF TABLES

549

the relation tP( - x) = 1 - tP(x). Values of x corresponding to a few special values of tP are given separately beneath the main table.

Table 3 Cumulative Chi-Square Distribution

This table gives values of u corresponding to a few selected values of F(u), where
F(u)

" f
o

x<n-2){2e-X/2 dx

2n/2r(n/2)

for n, the number of degrees of freedom, equal to 1, 2, ... , 30. For larger values of n, a normal approximation is quite accurate. The quantity V2u V2n 1 is nearly normal1y distributed with mean 0 and unit variance. Thus U/t, the exth quantile point of the distribution, may be computed by

where Zit is the exth quantile point of the cumulative normal distribution. As an ilJustra tion, we may compute the .95 value of U for n = 30 degrees of freedom:
U.95

,(1.645 + 1/59)2

43.S,

which is in error by Jess than 1 percent.

Table 4 Cumulative F Distribution

This table gives values of F corresponding to five values of

for selected values of m and n; m is the number of degrees of freedom in the numerator of F, and n is the number of degrees of freedom in the denominator of F. The table also provides values corresponding to G = .10, .OS, .025, .01, and .005 because F 1 - 1t for m and n degrees of freedom is the reciprocal of Fa for nand m degrees of freedom. Thus for G = .OS with three and six degrees of freedom, one finds
1 F.os(3, 6) = F.9s(6, 3) 1

= 8.94 = .112

One should interpolate on the reciprocals of m and n as in Table S for good accuracy.

550

TABLES

APPENDIX D

Table S Cumulative Students t Distribution

This table gives values of t corresponding to a few selected values of

with n = 1, 2, ... , 30, 40, 60, 120, 00. Since the density is symmetrical in t, it follows that F( - t) = 1 - F(t). One should not interpolate linearly between degrees of freedom but On the reciprocal of the degrees of freedom, if good accuracy in the last digit is desired. As an illustration, we shall compute the .975th quantile point for 40 degrees of freedom. The values for 30 and 60 are 2.042 and 2.000. Using the reciprocals of n, the interpolated value is
2.042 ao--Co
1 1 1 1

30-eo

(2.042 - 2.000) = 2.021,

which is the correct value.

Interpolating linearly, one would have obtained 2.028.

DESCRIPTION OF TABLES

551

Table 1 ORDINATES OF THE NORMAL DENSITY FUNCTION

.(x)

1 = -=

V21T

e- x 1./2

x
.0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9

.00

.01

.02 .3989 .3961 .3894 .3790 .3653 .3485 .3292 .3079 .2850 ,2613 .2371 .2131 .1895 .1669 .1456 .1257 .1074 .0909 .0761 .0632 .0519 .0422 .0339 .0270 .0213 .0167 .0129 .0099 .0075 .0056 .0042 .0031 .0022 .0016 .0012 .0008

.03 .3988 .3956 .3885 .3778 .3637 .3467 .3271 .3056 .2827 .2589 .2347 .2107 .1872 .1647 .1435 .1238 .1057 .0893 .0748 .0620 .0508 .0413 .0332 .0264 .0208 .0163 .0126 .0096 .0073 .0055 .0040 .0030 .0022 .0016 .0011 .0008 .0005 .0004 .0003 .0002

.04
.3986 .3951 .3876 .3765 .3621 .3448 .3251 .3034 .2803 .2565 .2323 .2083 .1849 .1626 .1415 .1219 .1040 .0878 .0734 .0608

.05 .3984 .3945 .3867 .3752 .3605 .3429 .3230 .3011 .2780 .2541 .2299 .2059 .1826 .1604 .1394 .1200 .1023 .0863 .0721 .0596

.06 .3982 .3939 .3857 .3739 .3589 .3410 .3209 .2989 .2756 .2516 .2275 .2036 .1804 .1582 .1374 .1182 .1006 .0848 .0707 .0584 .0478 .0387 .0310 .0246 .0194 .0151 .0116 .0088 .0067 .0050 .0037 .0027 .0020 .0014 .0010 .0007 .0005 .0003 .0002 .0002

.07 .3980 .3932 .3847 .3725 .3572 .3391 .3187 .2966 .2732 .2492 .2251 .2012 .1781 .1561 .1354 .1163 .0989 .0833 .0694 .0573 .0468 .0379 .0303 .0241 .0189 .0147 .0113 .0086 .0065

.08 .3977 .3925 .3836 .3712 .3555 .3372 .3166 .2943 .2709 .2468 .2227 .1989 .1758 .1539 .1334 .1145 .0973 .0818 .0681 .0562 .0459 .0371 .0297 .0235 .0184 .0143 .0110 .0084 .0063 .0047 .0035 .0025 .0018 .0013

.09 .3973 .3918 .3825 .3697 .3538 .3352 .3144 .2920 .2685 .2444 .2203 .1965 .1736 .15'18 .1315 .1127 .0957 .0804 .0669 .0551 .0449 .0363 .0290 .0229 .0180 .0139 .0107 .0081 .0061 .0046 .0034 .0025 .0018 .0013

.3989 .3989 .3970 .3965 .3910 .3902 ' .3802 .3814 .3683 .3668 .3521 .3332 .3123 .2897 .2661 .2420 .2179 .1942 .1714 .1497 .1295 .1109 .0940 .0790 .0656 .0540 .0440 .0355 .0283 .0224 .0175 .0136 .0104 .0079 .0060 .3503 .3312 .3101 .2874 .2637 .2396 .2155 .1919 .1691 .1476 .1276 .1092 .0925 .0775 .0644 .0529 .0431 .0347 0277 .0219 .0171 .0132 .0101 .0077 .0058 .0043 .0032 .0023 .0017 .0012 .0008

.0498 .0488 .0404 .0396 .0325 .0317 .0258 . .0252 .0203 .0198 .0158 .0122 .0093 .0071 .0053 .0039 .0029 .0021 .0015 .0011 .0008 .0005 .0004 .0003 .0002 .0154 .0119 .0091 .0069 .0051 .0038 .0028 .0020 .0015 .0010 .0007 ,0005 ,0004 .0002 .0002

.0048
.0036 .0026 .0019 .0014 .0010 .0007 .0005 .0003 .0002 .0002

.0044
.0033 .0024 .0017 .0012 .0009

.0009 .0007
.0005 .0003 .0002 .0001

.0009
.0006

.0006
.0004 .0003 .0002

.0006
.0004 .0003 .0002

.0006
.0004 .0003 .0002

,0004
.0003 .0002 .0001

552

TABLES

APPENDIX D

Table 2

CUMULATIVE NORMAL DISTRIBUTION

x
.0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 3.4

.00 .5000 .5398 .5793 .6179 .6554 .6915 .7257 .7580 .7881 .8159 .8413 .8643 .8849 .9032 .9192 .9332 .9452 .9554 .9641 .9713 .9772 .9821 .9861 .9893 .9918 .9938 .9953 .9965 .9974 .9981 .9987 .9990 .9993 .9995 .9997

.01 .5040 .5438 .5832 .6217 .6591 .6950 .7291 .7611 .7910 .8186 .8438 .8665 .8869 .9049 .9207 .9345 .9463 .9564 .9649 .9719 .9778 .9826 .9864 .9896
~9920

.02 .5080 .5478 .5871 .6255 .6628 .6985 .7324 .7642 .7939 .8212 .8461 .8686 .8888 .9066 .9222 .9357 .9474 .9573 .9656 .9726 .9783 .9830 .9868 .9898 .9922 .9941 .9956 .9967 .9976 .9982 .9987 .9991 .9994 .9995 .9997

.03 .5120 .5517 .5910 .6293 .6664 .7019 .7357 .7673 .7967 .8238 .8485 .8708 .8907 .9082 .9236 .9370 .9484 .9582 .9664 .9732 .9788 .9834 .9871 .9901 .9925 .9943 .9957 .9968 .9977 .9983 .9988 .9991 .9994 .9996 .9997

.04

.05 .5199 .5596 .5987 .6368 .6736 .7088 .7422 .7734 .8023 .8289 .8531 .8749 .8944 .9115 .9265 .9394 .9505 .9599 .9678 .9744 .9798 .9842 .9878 .9906 .9929 .9946 .9960 .9970 .9978 .9984 .9989 .9992 .9994 .9996 .9997

.06 .5239 .5636 .6026 .6406 .6772 .7123 .7454 .7764 .8051 .8315 .8554 .8770 .8962 . 9131 .9279 .9406 .9515 .9608 .9686 .9750 .9803 .9846 .9881 .9909 .9931 .9948 .9961 .9971 .9979 .9985 .9989 .9992 .9994 .9996 .9997

.07 .5279 .5675 .6064 .6443 .6808 .7157 .7486 .7794 .8078 .8340 .8577 .8790 .8980 .9147 .9292 .9418 .9525 .9616 .9693 .9756 .9808 .9850 .9884 .9911 .9932 .9949 .9962 .9972 .9979 .9985 .9989 .9992 .9995 .9996 .9997

.08 .5319 .5714 .6103 .6480 .6844 .7190 .7517 .7823 .8106 .8365 .8599 .8810

.09 .5359 .5753 .6141 .6517 .6879 .7224 .7549 .7852 .8133 .8389

.5160 .5557 .5948 .6331 .6700 .7054 .7389 .7704 .7995 .8264 .8508 .8729 .8925 .9099 .9251 .9382 .9495 .9591 .9671 .9738 .9793 .9838 .9875 .9904 .9927 .9945 .9959 .9969 .9977 .9984 .9988 .9992 .9994 .9996 .9997

.8621 .8830 .82~7 . .9015 .9162 .9177 .9306 .9319 .9429 .9535 :9625 .9699 .9761 .9812 .9854 .9887 .9913 .9934 .9951 .9963 .9973 .9980 .9986 .9990 .9993 .9995 .9996 .9997 .9441 .9545 .9633 .9706 .9767 .9817 .9857 .9890 .9916 .9936 .9952 .9964 .9974 .9981 .9986 .9990 .9993 .9995 .9997 .9998

.9940 .9955 .9966 .9975 .9982 .9987 .9991 .9993 .9995 .9997

x <I>(x)

1.282 .90 .20

1.645 .95 .10

1.960 .975 .05

2.326 .99 .02

2.576 .995 .01

3.090 .999 .002

3.291 .9995 .001

3.891 .99995 .0001

4.417 .999995 .00001

2[1 - <I>(x))

Table 3 CUMULATIVE CHI-SQUARE DISTRIBUTION

F(u) =

f:

x(n~ 2.)~2~- X/~ dx

~
1 2 3 4 5 6 7 8 9 10
11

.005 .04 393 .0100 .0717 .207 .412 .676 .989 1.34 1.73 2.16 2.60 3.07 3.57 4.07 4.60 5.14 5.70 6.26 6.84 7.43 8.03 8.64 9.26 9.89 10.5 11.2 11.8 12.5 13.1 13.8

.010 .0 3 157 .0201 .115 .297 .554 .872 1.24 1.65 2.09 2.56 3.05 3.57 4.11 4.66 5.23 5.81 6.41 7.01 7.63 8.26 8.90 9.54 10.2 10.9 11.5 12.2 12.9 13.6 14.3 15.0

.025 .0 3 982 .0506 .216 .484 .831 1.24 1.69 2.18

.050 .0 2 393 .103 .352 .711 1.15 1.64 2.17 2.73 3.33 3.94 4.57 5.23 5.89 6.57 7.26 7.96 8.67 9.39 10.1 10.9 11.6 12.3 13.1 13.8 14.6 15.4 16.2 16.9 17.7 18.5

.100 .0158 .211 .584 1.06 1.61 2.20 2.83 3.49 4.17 4.87 5.58 6.30 7.04 7.79 8.55 9.31 10.1 10.9 11.7 12.4 13.2 14.0 14.8 15.7 16.5 17.3 18.1 18.9 19.8 20.6

.250 .102 .575 1.21 1.92 2.67 3.45 4.25 5.07 5.90 6.74 7.58 8.44 9.30 10.2 11.0 11.9 12.8 13.7 14.6 15.5 16.3 17.2 18.1 19.0 19.9 20.8 21.7 22.7 23.6 24.5

.500 .455 1.39 2.37 3.36 4.35 5.35 6.35 7.34 8.34 9.34 10.3 11.3 12.3 13.3 14.3 15.3 16.3 17.3 18.3 19.3 20.3 21.3 22.3 23.3 24.3 25.3 26.3 27.3 28.3 29.3
-

.750 1.32 2.77 4.11 5.39 6.63 7.84 9.04 10.2 11.4 12.5 13.7 14.8 16.0 17.1 18.2 19.4 20.5 21.6 22.7 23.8 24.9 26.0 27.1 28.2 29.3 30.4 31.5 32.6 33.7 34.8
~-

.900

.950 .99 1* 7.81 9.49 11.1 12.6 14.1 15.5 16.9 18.3 19.7 21.0 22.4 23.7 25.0 26.3 27.6 28.9 30.1 31.4 32.7 33.9 35.2 36.4 37.7 38.9 40.1 41.3 42.6 43.8

.975 5.02 7.38 9.35 11.1 12.8 14.4 16.0 17.5 19.0 20.5 21.9 23.3 24.7 26.1 27.5 28.8 30.2 31.5 32.9 34.2 35.5 36.8 38.1 39.4 40.6 41.9 43.2 44.5 45.7 47.0

.990 6.63 9.21 11.3 13.3 15.1 16.8 18.5 20.1 21.7 23.2 24.7 26.2 27.7 29.1 30.6 32.0 33.4 34.8 36.2 37.6 38.9 40.3 41.6 43.0 44.3 45.6 47.0 48.3 49.6 50.9

.995 7.88 10.6 12.8 14.9 16.7 18.5 20.3 22.0 23.6 25.2 26.8 28.3 29.8 31.3 32.8 34.3 35.7 37.2 38.6 40.0 41.4 42.8 44.2 45.6 46.9 48.3 49.6 51.0 52.3 53.7

2.10,
3.25 . 3.82 4.40 5.01 5.63 6.26 6.91 7.56 8.23 8.91 9.59 10.3 11.0 11.7 12.4 13.1 13.8 14.6 15.3 16.0 16.8

12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

2.71 4.61 6.25 7.78 9.24 10.6 12.0 13.4 14.7 16.0 17.3 18.5 19.8 21.1 22.3 23.5 24.8 26.0 27.2 28.4 29.6 30.8 32.0 33.2 34.4 35.6 36.7 37.9 39.1 40.3
--

!...oJ

v. v.

This table is abridged from "Tables of percentage points of the incomplete beta function and of the chi-square distribution," Biometrika, Vol. 32 (1941). It is here published with the kind permission of its author, Catherine M. Thompson, and the editor of Biometrika.

Table 4

CUMULATIVE F DISTRIBUTION (m degrees of freedom in numerator; n in denominator)

F

G(F) =

fo
4

r(m +
2

n)m"'12 rf I2 x <m-2)/2(n

+ mx)-(m+n)/2dx

..

G
.90 .95 .975 .99 .995 .90 .95 .975 .99 .995 .90 .95 .975 .99 .995 .90 .95 .975 .99 .995 .90 .95 .975 .99 .995 .90 .95 .975 .99 .995 .90 .95 .975 .99 .995 .90 .95 .975 .99 .995

10

12

15

20

30

60

120

00

49.5 53.6 39.9 58.9 55.8 57.2 58.2 59.4 59.9 60.2 60.7 61.2 61.7 62.3 62.8 63.1 63.3 161 200 216 237 225 230 234 239 241 244 242 246 248 250 252 253 254 648 800 864 948 900 922 937 957 963 969 977 985 993 1000 1010 1010 1020 4,050 5,000 5,400 5,620 5,760 5,860 5,930 5,980 6,020 6,060 6,110 6,160 6,210 6,260 6,310 6,340 6,370 16,200 20,000 21,600 22,500 23,100 23,400 23,700 23,900 24,100 24,200 24,400 24,600 24,800 25,000 25,200 25,400 25,500 8.53 18.5 38.5 98.5 199 5.54 10.1 17.4 34.1 55.6 4.54 7.71 12.2 21.2 31.3 4.06 6.61 10.0 16.3 22.8 3.78 5.99 8.81 13.7 18.6 3.59 5.59 8.07 12.2 16.2 3.46 5.32 7.57 11.3 14.7 9.00 19.0 39.0 99.0 199 5.46 9.55 16.0 30.8 49.8 4.32 6.94 10.6 18.0 26.3 3.78 5.79 8.43 13.3 18.3 3.46 5.14 7.26 10.9 14.5 3.26 4.74 6.54 9.55 12.4 3.11 4.46 6.06 8.65 11.0 9.16 19.2 39.2 99.2 199 5.39 9.28 15.4 29.5 47.5 4.19 6.59 9.98 16.7 24.3 3.62 5.41 7.76 12.1 16.5 3.29 4.76 6.60 9.78 12.9 3.07 4.35 5.89 8.45 10.9 2.92 4.07 5.42 7.59 9.60 9.24 19.2 39.2 99.2 199 5.34 9.12 15.1 28.7 46.2 4.11 6.39 9.60 16.0 23.2 3.52 5.19 7.39 11.4 15.6 3.18 4.53 6.23 9.15 12.0 2.96 4.12 5.52 7.85 10.1 2.81 3.84 5.05 7.01 8.81 9.29 19.3 39.3 99.3 199 5.31 9.01 14.9 28.2 45.4 4.05 6.26 9.36 15.5 22.5 3.45 5.05 7.15 11.0 14.9 3.11 4.39 5.99 8.75 11.5 2.88 3.97 5.29 7.46 9.52 2.73 3.69 4.82 6.63 8.30 9.33 19.3 39.3 99.3 199 5.28 8.94 14.7 27.9 44.8 4.01 6.16 9.20 15.2 22.0 3.40 4.95 6.98 10.7 14.5 3.05 4.28 5.82 8.47 11.1 2.83 3.87 5.12 7.19 9.16 2.67 3.58 4.65 6.37 7.95 9.35 19.4 39.4 99.4 199 5.27 8.89 14.6 27.7 44.4 3.98 6.09 9.07 15.0 21.6 3.37 4.88 6.85 10.5 14.2 3.01 4.21 5.70 8.26 10.8 2.78 3.79 4.99 6.99 8.89 2.62 3.50 4.53 6.18 7.69 9.37 19.4 39.4 99.4 199 5.25 8.85 14.5 27.5 44.1 3.95 6.04 8.98 14.8 21.4 3.34 4.82 6.76 10.3 14.0 2.98 4.15 5.60 8.10 10.6 2.75 3.73 4.90 6.84 8.68 2.59 3.44 4.43 6.03 7.50 9.38 19.4 39.4 99.4 199 5.24 8.81 14.5 27.3 43.9 3.93 6.00 8.90 14.7 21.1 3.32 4.77 6.68 10.2 13.8 2.96 4.10 5.52 7.98 10.4 2.72 3.68 4.82 6.72 8.51 2.56 3.39 4.36 5.91 7.34 9.39 19.4 39.4 99.4 199 5.23 8.79 14.4 27.2 43.7 3.92 5.96 8.84 14.5 21.0 3.30 4.74 6.62 10.1 13.6 2.94 4.06 5.46 7.87 10.2 2.70 3.64 4.76 6.62 8.38 2.54 3.35 4.30 5.81 7.21 9.41 19.4 39.4 Q9.4 199 5.22 8.74 14.3 27.1 43.4 3.90 5.91 8.75 14.4 20.7 3.27 4.68 6.52 9.89 13.4 2.90 4.00 5.37 7.72 10.0 2.67 3.57 4.67 6.47 8.18 2.50 3.28 4.20 5.67 7.01 9.42 19.4 39.4 99.4 199 5.20 8.70 14.3 26.9 43.1 3.87 5.86 8.66 14.2 20.4 3.24 4.62 6.43 9.72 13.1 2.87 3.94 5.27 7.56 9.81 2.63 3.51 4.57 6.31 7.97 2.46 3.22 4.10 5.52 6.81 9.44 19.5 39.4 99.4 199 5.18 8.66 14.2 26.7 42.8 3.84 5.80 8.56 14.0 20.2 3.21 4.56 6.33 9.55 12.9 2.84 3.87 5.17 7.40 9.59 2.59 3.44 4.47 6.16 7.75 2.42 3.15 4.00 5.36 6.61 9.46 19.5 39.5 99.5 199 5.17 8.62 14.1 26.5 42.5 3.82 5.75 8.46 13.8 19.9 3.17 4.50 6.23 9.38 12.7 2.80 3.81 5.07 7.23 9.36 2.56 3.38 4.36 5.99 7.53 2.38 3.08 3.89 5.20 6.40 9.47 19.5 39.5 99.5 199 5.15 8.57 14.0 26.3 42.1 3.79 5.69 8.36 13.7 19.6 3.14 4.43 6.12 9.20 12.4 2.76 3.74 4.96 7.06 9.12 2.51 3.30 4.25 5.82 7.31 2.34 3.01 3.78 5.03 6.18 9.48 19.5 39.5 99.5 199 5.14 8.55 13.9 26.2 42.0 3.78 5.66 8.31 13.6 19.5 3.12 4.40 6.07 9.11 12.3 2.74 3.70 4.90 6.97 9.00 2.49 3.27 4.20 5.74 7.19 2.31 2.97 3.73 4.95 6.06 9.49 19.5 39.5 99.5 199 5.13 8.53 13.9 26.1 41.8 3.76 5.63 8.26 13.5 19.3 3.11 4.37 6.02 9.02 12.1 2.72 3.67 4.85 6.88 8.88 2.47 3.23 4.14 5.65 7.08 2.29 2.93 3.67 4.86 5.95

.90 .95 .975 .99 .995 .90 .95 .975 .99 .995 .90 .95 .975 .99 .995 .90 .95 .975 .99 .995 .90 .95 .975 .99 .995

3.36 5.12 7.21 10.6 13.6 3.29 4.96 6.94 10.0. 12.8 3.18 4.75 6.55 9.33 11.8 3.07 4.54 6.20 8.68 10.8 2.97 4.35 5.87 8.10 9.94 2.88 4.17 5.57 7.56 9.18 2.79 4.00 5.29 7.08 8.49 2.75 3.92 5.15 6.85 8.18 2.71 3.84 5.02 6.63 7.88

3.01 4.26 5.71 8.02 10.1 2.92 4.10 5.46 7.56 9.43 2.81 3.89 5.10 6.93 8.51 2.70 3.68 4.77 6.36 7.70 2.59 3.49 4.46 5.85 6.99 2.49 3.32 4.18 5.39 6.35 2.39 3.15 3.93 4.98 5.80 2.35 3.07 3.80 4.79 5.54 2.30 3.00 3.69 4.61 5.30

2.81 3.86 5.08 6.99 8.72 2.73 3.71 4.83 6.55 8.08 2.61 3.49 4.47 5.95 7.23 2.49 3.29 4.15 5.42 6.48 2.38 3.10 3.86 4.94 5.82 2.28 2.92 3.59 4.51 5.24 2.18 2.76 3.34 4.13 4.73 2.13 2.68 3.23 3.95 4.50 2.08 2.60 3.12 3.78 4.28

2.69 3.63 4.72 6.42 7.96 2.61 3.48 4.47 5.99 7.34 2.48 3.26 4.12 5.41 6.52 2.36 3.06 3.80 4.89 5.80 2.25 2.87 3.51 4.43 5.17 2.14 2.69 3.25 4.02 4.62 2.04 2.53 3.01 3.65 4.14 1.99 2.45 2.89 3.48 3.92 1.94 2.37 2.79 3.32 3.72

2.61 3.48 4.48 6.06 7.47 2.52 3.33 4.24 5.64 6.87 2.39 3.11 3.89 5.06 6.07 2.27 2.90 3.58 4.56 5.37 2.16 2.71 3.29 4.10 4.76 2.05 2.53 3.(\3 3.70 4.23 1.95 2.37 2.79 3.34 3.76 1.90 2.29 2.67 3.17 3.55 1.85 2.21 2.57 3.02 3.35

2.55 3.37 4.32 5.80 7.13 2.46 3.22 4.07 5.39 6.54 2.33 3.00 3.73 4.82 5.76 2.21 2.79 3.41 4.32 5.07 2.<)9 2.60 3.13 3.87 4.47 1.98 2.42 2.87 3.47 3.95 1.87 2.25 2.63 3.12 3.49 1.82 2.18 2.52 2.96 3.28 1.77 2.10 2.41 2.80 3.09

2.51 3.29 4.20 5.61 6.88 2.41 3.14 3.95 6.30

2.47 3.23 4.10 5.47 6.69 2.38 3.07 3.85 5.06 6.12 2.24 2.85 3.51 4.50 5.35 2.12 2.64 3.20 4.00 4.67 2.00 2.45 2.91 3.56 4.()9 1.88 2.27 2.65 3.17 3.58 1.77 2.10 2.41 2.82 3.13 1.12 2.02 2.30 2.66 2.93 1.67 1.94 2.19 2.51 2.74

2.44 3.18 4.03 5.35 6.54 2.35 3.02 3.78 4.94 5.97 2.21 2.80 3.44 4.39 5.20 2.09 2.59 3.12 3.89 4.54 1.96 2.39 2.84 3.46 3.96 1.85 2.21 2.57 3.07 3.45 1.74 2.04 2.33 2.72 3.01 1.68 1.96 2.22 2.56 2.81 1.63 1.88 2.11 2.41 2.62

2.42 3.14 3.96 5.26 6.42 2.32 2.98 3.72 4.85 5.85 2.19 2.75 3.37 4.30 5.<)9 2.06 2.54 3.06 3.80 4.42 1.94 2.35 2.77 3.37 3.85 1.82 2.16 2.51 2.98 3.34 1.71 1.99 2.27 2.63 2.90 1.65 1.91 2.16 2.47 2.71 1.60 1.83 2.05 2.32 2.52

2.38 3.07 3.87 5.11 6.23 2.28 2.91 3.62 4.71 5.66 2.15 2.69 3.28 4.16 4.91 2.02 2.48 2.96 3.67 4.25 1.89 2.28 2.68 3.23 3.68 1.77 2.09 2.41 2.84 3.18 1.66 1.92 2.17 2.50 2.74 1.60 1.83 2.05 2.34 2.54 1.55 1.75 1.94 2.18 2.36

2.34 3.01 3.77 4.96 6.03 2.24 2.84 3.52 4.56 5.47 2.10 2.62 3.18 4.01 4.72 1.97 2.40 2.86 3.52 4.07 1.84 2.20 2.57 3.09 3.50 1.72 2.01 2.31 2.70 3.01 1.60 1.84 2.06 2.35 2.57 1.54 1.75 1.94 2.19 2.37 1.49 1.67 1.83 2.04 2.19

2.30 2.94 3.67 4.81 5.83 2.20 2.77 3.42 4.41 5.27 2.06 2.54 3.07 3.86 4.53 1.92 2.33 2.76 3.37 3.88 1.79 2.12 2.46 2.94 3.32 1.67 1.93 2.20 2.55 2.82 1.54 1.75 1.94 2.20 2.39 1.48 1.66 1.82 2.03 2.19 1.42 1.57 1.71 1.88 2.00

2.25 2.86 3.56 4.65 5.62 2.15 2.70 3.31 4.25 5.07 2.01 2.47 2.96 3.70 4.33 1.87 2.25 2.64 3.21 3.69 1.74 2.04 2.35 2.78 3.12 1.61 1.84 2.07 2.39 2.63

2.21 2.79 3.45 4.48 5.41 2.11 2.62 3.20 4.08 4.86 1.96 2.38 2.85 3.54 4.12 1.82 2.16 2.52 3.05 3.48 1.68 1.95 2.22 2.61 2.92 1.54 1.74 1.94 2.21 2.42

2.18 2.75 3.39 4.40 5.30 2.08 2.58 3.14 4.00 4.75 1.93 2.34 2.79 3.45 4.01 1.79 2.1 J. 2.46 2.96 3.37 1.64 1.90 2.16 2.52 2.81 1.50 1.68 1.87 2.11 2.30 1.35 1.47 1.58 1.73 1.83 1.26 1.35 1.43 1.53 1.61 1.17 1.22 1.27 1.32 1.36

2.16 2.71 3.33 4.31 5.19 2.06 2.54 3.08 3.91 4.64 1.90 2.30 2.72 3.36 3.90 1.76 2.07 2.40 2.87 3.26 1.61 1.84 2.09 2.42 2.69 1.46 1.62 1.79 2.01 2.18 1.29 1.39 1.48 1.60 1.69 1.19 1.25 1.31 1.38 1.43 1.00 1.00 1.00 1.00 1.00

10

~.20

12

2.28 2.91 3.61 4.64 5.52 2.16 2.71 3.29 4.14 4.85 2.04 2.51 3.0t 3.70 4.26 1.93 2.33 2.75 3.30 3.74 1.82 2.17 2.51 2.95 3.29 1.77 2.09 2.39 2.79 3.09 1.72 2.01 2.29 2.64 2.90

15

20

.90 .95 .975 .99 .995

.90 .95 .975 .99 .995 .90 .95 .975 .99 .995 .90 .95 .975 .99 .995

30

60

1.48 . 1.40 1.65 1.53 1.82 1.67 2.03 1.84 2.19 1.96 1.41 1.55 1.69 1.86 1.98 1.34 1.46 1.57 1.70 1.79 1.32 1.43 1.53 1.66 1.75 1.24 1.32 1.39 1.47 1.53

120

>

* This table is abridged from "Tables of percentage points of the inverted beta distribution," Biometrika, Vol. 33 (1943). It is here published with the kind permission of its authors, Maxine Merrington and Catherine M. Thompson, and the editor of Biometrika.

556

TABLES

APPENDIX 0

Table 5

CUMULATIVE STUDENT'S t DISTRIBUTION*

,
Fr.t)

r(n11) r(n/2)V -rrn

=f

- <:D

_ (1 + x'),O+'>l _
n
.975 12.706 4.303 3.182 1.776 2.571 2.447 2.365 2.306 2.262 2.228 2.201 2.179 2.160 2.145 2.131 2.120 2.110 2.101 2.093 2.086 2.080 2.074 2.069 2.064 2.060 2.056 2.052 2.048 2.045 2.042 2.021 2:000 1.980 1.960

dx

~
1 2 3 4 5 6 7 8 9 10

.75 1.000 .816 .765 .741 .727 .718 .711 .706 .703 .700 .697 .695 .694 .692 .691 .690 .689 .688 .688 .687 .686 .686 .685 .685 .684 .684 .684 .683 .683 .683 .681 .679 .677 .674

.90 3.078 1.886 1.638 1.533 1.476 1.440 1.415 1.397 1.383 1.372 1.363 1.356 1.350 1.345 1.341 1.337 1.333 1.330 1.328 1.325 1.323 1.321 1.319 1.318 1.316 1.315 1.314 1.313 1.311 1.310 1.303 1.296 1.289 1.282

.95 6.314 2.920 2.353 2.132 2.015 1.943 1.895 1.860 1.833 1.812 1.796 1.782 1.771 1.761 1.753 1.746 1.740 1.734 1.729 1.725 1.721 1.717 1.714 1.711 1.708 1.706 1.703 1.701 1.699 1.697 1.684 1.671 1.658 1.645

.99 31.821 6.965 4.541 3.747 3.365 3.143 2.998 2.896 2.821 2.764 2.718 2.681 2.650 2.624 2.602 2.583 2.567 2.552 2.539 2.528 2.518 2.508 2.500 2.492 2.485 2.479 2.473 2.467 2.462 2.457 2.423 2.390 2.358 2.326

.995 63.657 9.925 5.841 4.604 4.032 3.707 3.499 3.355 3.250 3.169 3.106 3.055 3.012 2.977 2.947 2.921 2.898 .2878 2.861 2.845 2.831 2.819 2.807 2.797 2.787 2.779 2.771 2.763 2.756 2.750 2.704 2.660 2.617 2.576

.9995 636.619 31.598 12.941 8.610 6.859 5.959 5.405 5.041 4.781 4.587 4.437 4.318 4.221 4.140 4.073 4.015 3.965 3.922 3.883 3.850 3.819 3.792 3.767 3.745 3.725 3.707 3.690 3.674 3.659 3.646 3.551 3.460 3.373 3.291

11
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 40 60 120
00

This table is abridged from the" Statistical Tables n of R. A. Fisher and Frank Yates published by Oliver &, Boyd, Ltd., Edinburgh and London, 1938. It is here pubHshed with the kind permission of the authors and their publishers.
III

INDEX
Absolutely continuous, 60, 61, 63,64 Admissible estimator, 299 Algebra of sets, 18, 22 Analysis of variance, 431 A posteriori probability, 5, 9 A priori probability, 2-4, 9 Arithmetic series, 528 Asymptotic distribution, 196,256-258,261,359, 440,444 Average sample size in sequential tests, 410--472 BAN estimators, 294, 296, 349, 446 Bayes estimation, 339, 344 Bayes' formula, 36 Bayes risk, 344 Bayes test, 411 Bayesian interval estimates, 396,391 BernouUi distribution, 81, 538 Bernoulli trial, 88 repeated independent, 89,101,131 Best linear unbiased estimators, 499 Beta distribution, 115, 540 of second kind, 215 Beta function, 534, 535 Bias, 293 Binomial coefficient, 529 Binomial distribution, 81-89, 119, 120,538 confidence limits for p, 393, 395 normal approximation, 120 Poisson approximation, 119 Binomial theorem, 530 Birthday problem, 45 Bivariate normal distribution, 162-168 conditional distribution, 161 marginal distribution, 161 moment generating function, 164 moments, 165 Boole's inequality, 25 Cauchy distribution, 111, 201, 238, 540 Cauchy Schwarz inequalities, 162 Censored, 104 Central-limit theorem, Ill, 120, 195, 233, 234, 258 Centroid, 65 Chebyshev inequality, 11 multivariate, 172 Chi-square distribution, 241,542,549 table of, 553 Chi-square tests, 440, 442-461 contingency tables, 452-461 goodness-of-fit, 442, 441 Combinations and permutations, 528 Combinatorial symbol, 528 Complement of set, 10 Complete families of densities, 321, 324, 354 Complete statistic, 324 Completeness, 321, 354 Composite hypothesis, 418 (See also Hypotheses) Concentration, 289 Conditional distributions, 129, 148 bivariate normal, 161 continuous, 146, 141 discrete, 143-145 Conditional expectation, 151 Conditional mean, 158 Conditional probability, 32 Conditional variance, 159 Confidence bands for c,d.f., 511 Confidence coefficient. 315,311,461 Confidence intervals, 313, 315, 311, 461 c,d,f.,511 difference in means, 386 general method for, 389 large sample, 393 mean of normal population, 315,381,384 median, 512 method of finding tests, 425, 461 on~sided, 318 p of binomial population, 393, 395 pivotal method of obtaining, 319, 381 regression coefficients, 491-494 uniformly most accurate, 464 variance of normal population, 382, 384 Confidence limits [see Confidence interval(s)]
Confidencere~on,311

for mean and variance of normal population, 384 Confidence sets, 461 uniformly most accurate, 464 Consistency of an estimator, 291, 294, 295. 359 Contagious distribution, 102, 122, 123 Contingency tables, 452-461 interaction, 454 tests for independence, 452 Continuous distributions, 60, 62 (See also Distributions) Continuous random variable, 60 Convex function, 12 Convolution, 186 Correlation, 155, 161 sample, 526 Spearman's rank, 525, 526 Correlation coefficient, 155, 156 Covariance, 155, 156 of two Bnear combinations of random variables, 119 Covariance matrix, 352, 489 Cramer-Rao inequality, 316 Cramer-Rao lower bound, 316, 320 Critical function, 404

SS8

INDEX

Critical region, 403 size of, 407 Cumulant generating function, 80 Cumulants, 80 Cumulative distribution function(s), 54, 56, 63, 130,132,144 bivariate, 132 decomposition of, 63 empirical,264, 506 joint, 130 sample, 264,287,506,511 unidimensional, 54, 56

Distributions: degenerate, 258 discrete, 58, 60 discrete uniform, 86, 538 double exponential, 117, 540 exponential, 111, 121,237,262,540

F,246,247,542,549,554
gamma, 111, 123, 540 geometric, 99, 538 Gumbel, 118,542 hypergeometric, 91, 538 joint, 130, 133, 138 lambda, 128 Laplace, 117 limiting, 196,258,261,444,446,507-509 linear function of normal variates, 194 logarithmic, 105 logistic, 118,540 lognormal, 117, 540 marginal, 132, 135, 141 MaxwelJ, 127 multinomial, 137 covariance for, 196 multivariate, 129-174 negative binomial, 99, 538 negative hypergeometric, 213 normal (see Normal distribution) order statistics, 251, 254 Pareto, 118, 542 Pearsonian system, 118-119 Poisson, 93,104,119-121,123,236,538 prior, 340, 417 r distribution, 127 Rayleigh, 127 rectangular, 105,238,540 sample, 224 Student's t, 249, 250,542,556 symmetric, 170 table of, 538-543 truncated, 122 Tukey's symmetrical lambda, 128 uniform, 105,238,540 continuous, 105, 540 discrete, 86, 538 variance ratio, 246,437,438 Weib"lll, 117,542 (See alsQ Sampling distributions) Efficiency, 291 EJIipsoid of concentration, 353 Empty set, 10 Equivalent sets, 10 Error: mean-squared,291 size of, 405 Type I, 405 Type II, 405

Decision theory, 297,343,415 Deductive inference, 220 Degenerate distribution, 258 Degrees of freedom, 242, 246 De Moivre-Laplace limit theorem, 120 De Morgan's law, 13,14 Density functions, 62 discrete, 57, 135 joint discrete, 133 joint probability, 138 probability, 60, 62 (See also Distributions) Difference between means: confidence intervals, 386 tests, 432 Differentiation, 532 Discrete distributions, 57, 133, 144, 145 (See also Distributions) Discrete random variables, 57, 63, 133 Disjoint, 14 Distance function, 287 Distribution-free tests, 505,509,514,518-524 Distribution functions, 54,56,63, 130, 132 of difference, 185 of maximum, 182 of minimum, 182 of order statistics, 251 of product, 187 of quotient, 187 of sums, 185 Distributions: asymptotic, 196,256-264,359,440,444 BernouJJi, 87, 236,538 beta, 115, 540 beta-binomial, 104 binomial (see Binomial distribution) bivariate normal, 162-168 Cauchy, 117,207,238,540 chi-square, 241,542,549 conditional (see Conditional distributions) contagious, 102,122,123 continuous, 60, 62 cumulative [see Cumulative distribution function (s) ]

INDEX

559

Estimation: interval, 372 point, 211 Estimator ( s) ,212, 213 admissible, 299 BAN, 294, 296 Bayes method, 286, 339, 344 best linear unbiased, 499 better, 299 closeness, 288 concentrated,289 consistent, 294, 295, 359 e1Iipsoid of concentration, 353 least squares, 286,498 location invariant, 334 maximum likelihood (see Maximum likeHhood estimators) mean-squared error, 291 method of moments, 214 minimax, 299,350 minimum chi-square, 286, 281 minimum distance, 286, 281 Pitman, 290, 334, 336 scale invariant, 336 unbiased, 293,315 uniformly minimum-variance unbiased, 315 Wilks' generalized variance, 353 (See also Large samples) Event, 14, 15, 18,53 elementary, 15 Event space, 15, 18,23 Excess, coefficient of, 16 Expectation, 64, 69,153,160,116 Expected values, 64, 69, 10, 129,153,160 conditional, 151 of functions of random variables, 116 properties of, 10 Exponential c1ass, 312, 313,320,326,355,422 Exponential distribution, 111,121,231,262,540 Exponential family, 312, 313,320, 326, 355, 422 Extension theorem, 22 Extreme-value statistic, 118,258 asymptotic distribution of, 261
F distribution, 246, 542, 549

Function: decision, 291 definition of, 19 density (see Distributions) distance, 281 distribution (see Distributions) domain of, 19,53 gamma, 534 generating, 84 image of, 19 indicator, 20 likelihood,218 loss, 291 squared-error, 291 moment generating (see Moment generating function) power, 406 preimage, 19 probability, 21,22,26 regression, 158, 168 risk, 291, 298 set, 20, 21 size-of-set, 21

Game of craps, 48 Gamma distribution, 111, 112, 123, 540 Gamma function, 534 Gauss-Markov theorem, 500 Gaussian distribution (see Normal distribution) Generalized likelihood ratio (see Likelihood ratio) Generalized variance, 352, 353 Generating functions (see specific generating functions) Geometric distribution, 99, 538 Geometric series, 528 G1ivenko-Cantelli theorem, 501 Goodness-of-fit test: chi-si\uare, 442, 441 Kolmogorov-Smirnov, 508, 509 Gumbel distribution, 118, 542

table of, 554 Factorial moment generating function, 19 Factorial moments, 11, 19 Factorial notations, 528 Factorial symbol, 528 Factorization criterion, 301 Finite population, sampling from, 261 Frequency function, 58 Function, 19 beta, 535 convex, 12 counterdomain of, 19, 53

Homogeneity of populations, test of, 505 two exponentials, 416 two multinomials, 450 two normals, 432, 435 two Poissons, 451 two trinomials, 419 Homogeneity of variances, test of, 438,439 Hypergeometric distributions, 91, 538 Hypotheses, statistical: alternative, 405 composite, 402 null, 405 simple. 402, 409 (See also Tests of hypotheses)

560

INDEX

Ideal power function, 406 Incomplete beta function, 115,513, 516 Incomplete gamma function, 114 Independence, 32, 150,160 in contingency tables, 452 of events, 40, 41, 46 in probability sense, 32,143, 150, 160,161 of random variables, 150 of sample mean and variance, 243, 245 stochastic, 150 Index set, 13 Inference, 220, 271 deductive, 220 inductive, 220,221 Information, 300, 301 Interquartile range, 75 Intersection, 10, 13 Interval estimation, 372 Bayesian, 396 large sample, 393 (See also Confidence intervals) Invariance: location, 331~336 of maximum likelihood estimators, 284, 285, 442 scale, 331, 336-338 Inverse binomial sampling, 103

Likelihood ratio: generalized,419 large-sample distribution for, 440 monotone, 423, 424 simple, 410, 423 tests, 409, 419, 440 Limiting distribution, 196,258,261,444,446, 507-509 Linear function of normal variates, distribution of, 194 Linear models, 482, 485 confidence intervals, 491 point estimation, 487, 498 tests of hypotheses, 494 Linear regression, 482 Location invariance, 332~336 Location parameter, 333 Logistic, 118,540 Lognormal distribution, 117, 540 Lossfunction,297,343,414,415

Jacobian, 205 Jensen inequality, 72 Joint distributions, 129ff, Joint moments, 159

Kolmogorov-Smirnov goodness-of-fit test, 508-510 Kurtosis, 76

Lagrange multipliers, 501 Large sample, 294, 358 confidence limits, 393 distribution of estimators, 359 distribution of generalized likelihood ratio, 440 distribution of mean, 233 Law of large numbers, 231, 232,258 Least squares, 498 Lehmann-Scheffe theorem, 326, 356 Leibniz' rule, 532 I'Hospital's rule, 532 Liars problem, 47 Likelihood function, 278 induced, 285 Likelihood ratio: exact "distribution of, 480

Marginal distributions: for bivariate normal di~tribution, 167 continuous, 141 discrete, 132, 135 Mass, 58 Mass point, 58 Maximum of random variables, 182 Maximum likelihood, principle of, 276 Maximum-likelihood estimators, 279 invariance property, 284,285 large-sample distribution of, 358, 359 of parameters: of normal distribution, 281 of uniform distribution, 282 properties of, 284, 358 Mean: definition of, 64 distribution of, 236-238 sample, 228, 230 variance of, 231 Mean absolute deviation, 297 Mean-squared error, 291, 297 Mean-squared-error consistency, 294, 295 Median, 73,255 sample, 255 tests of, 521 Mendelian inheritance,445 Method of moments, 274 Midrange, 255 ,. Minimal sufficient statistic, 311 Minimax estimator, 299,350 Minimax test, 416 Minimum of random variables, 182 Minimum chi-square estimation, 286, 287 Minimum distance estimation, 286, 287

INDEX

561

Minimum-variance unbiased estimator, uniformly (UMVUE), 315 Mixture, 122, 123 Mode, 74 Model: functional, 483 Jinear, 483,485 Moment generating function, 72, 78, 80, 159161,164 factorial, 79 of random variables, 159, 160 table of, 538-543 Moment problem, 81 Moments, 64, 72, 73 central, 73 cumulant, 80 estimators of, 227 factorial, 77 joint, 159 population, 227 problem of, 81 raw, 72, 73 sample, 227 Monotone likelihood ratio, 423, 424 Multinomial distribution, 137,253,443 covariance of, ] 96 tests on, 448 Multinomial theorem, 530, 531 Multiplication rule, 37 Multivariate distributions, 129ft marginal and conditional distributions for, 132,135,141,143-147 moment generating function for, 160 moments of~ 159 Mutua]]yexclusive, 3, 14,41

Normal distribution, 107-111, 120,239,540,548 bivariate, 162, 525 conditional, 167 independence of sample mean and variance, 243,245 marginal, 167 moment generating function for, 164 multivariate, 162 regression functions for, 168 role of, 239 sample mean, 240 sample variance, 241 table of, 552 truncated, 124 Normal equations, 487 Null hypothesis (see Hypotheses, statistical)

Order statistics, 251 asymptotic distribution of, 256-264 distribution of functions of, 254 Parameter, 85 parameter space, 273, 351 Parametric family, 85 Pareto distribution, 118, 542 Partition, 300 Pearson's chi-square tests, 444, 459,461 Pitman-closer, 290 Pitman estimator for location, 334 Pitman estimator for scale, 337 Pivotal quantity, 379 Pivotal quantity method, 379, 387 Poisson distribution, 93, 104, 119-121, 123,236, 538 compound, 123 Populations, 222, 224 sampled, 223 target, 222 Posterior Bayes estimator, 341 Posterior distribution, 340 Posterior risk, 346 Power function of test, 406, 411 ideal, 406 Prior distribution, 340,417 ProbabiJity,2 a posteriori, 5, 9 a priori, 2-4, 9 axioms of. 8, 22 classical, 2, 3, 5 conditional, 32, 42 properties of, 34 definition, 3, 21, 22 equa]]y likely, 3, 5, 25 frequency, 2, 5, 6 function, 21, 22

Negative binomial distribution, 99, 438 Negative hypergeometric distribution, 213 Neyman~Pearson lemma, 411 Nonparametric methods, 504 confidence intervals, 512 equality of distributions, 518-524 interval estimates, 512 Kolmogorov-Smirnov statistic, 508-510 median, 512 median test, 521 point estimation, 512 quantiles, 512 rank correlation, 525 rank-sum test, 522 run'test, 519 sign test: one-sample. 514 two-sample. 519 tests (see Tests of hypotheses) tolerance limits, 515

56.2

INDEX

Probability: independence, 32,40-42 integral transformation, 202 laws of, 23-25,34-37 mass function, 58 models, 8 properties of, 23-25,34-37 space, 25, 53 subjective, 9 total, 35 Probability density function, 57, 60,62,141 Probability function, 58 equal1y likely, 26 Probability generating function, 64 Probability integral transform, 202 Problem of moments, 81 Product of two random vaTiables, 180 Product notation, 527 Propagation of errors, 181

Regression function (see Regression curve) Relative frequency, 3, 6, 8 Reparameterize, 441, 442 Risk: Bayes, 344 function,297,298,415 posterior, 346 Runs, 519

Quantile, 73 point and interval estimates, 512 tests of hypotheses, 514 Quotient of two random variables, 180 Random number, 107,202 of random variables, 197 Random sampling, 223 Random variables, 53 continuous, 60, 138 definition of, 53 discrete, 57, 133 joint, 133, 138 maximum of, 182 minimum of, 182 mixed,62 product, 180 quotient, 180 sums, 178 (See also Distributions) Range, 255 of function, 19 interquartile, 75 of sample, 255 Rank-sum test, 522 Rao-B1ackwell theorem, 321,354 Rectangular distribution, 105,238, 540 References, 544 Regression, linear, 482 Regression coefficient: confidence interval, 491 estimators, 491 tests, 494 Regression curve, 158 for normal, 168

Sample, 222 cumulative distribution function, 264 distribution of, 224 mean, 227,228, 230 median, 255 midrange, 255 moments, 226, 227 quantiles, 251 random, 223 range, 255 variance, 229, 245 Sample c.d,f., 264 interferences on, 506 Sample mean, 219, 227,228, 230, 240 variance of, 231 Sample moments, 219,227 Sample point, 9 Sample space, 9, 14 finite, 15, 31, 25ff. Sample variance, 229, 245 Sampled populations, 223 Sampling, 27,219 with replacement, 27 without replacement, 27 Sampling distributions, 219, 224 for difference of two means, 386 for mean: from binomial population, 236 from Cauchy population, 238 from exponential, 237 of large samples, 232-235 from normal, 241 Poisson population, 236 from uniform population, 238 for order statistics, 251ff. for ratio of sample variances, 247 for regression coefficients, 490 Scale invariance, 336-338 Scale parameter, 336 Semiinvariants, 80 Sequential probability ratio test, 464, 466-467 approximate, 468 expected sample size of, 468, 470 Sequential 'tests, 464 for binomial, 481 fundamental identity for, 470

INDEX

563

Sequential tests: for mean of normal population, 471 sample size in, 468 Set, 9 complements, 10 confidence, 461 difference, 10 disjoint, 14 empty, 10 equivalence of, 10 function, 21 index, 13 intersection, 10, 13 laws of, 11 mutually exclusive, 14 null,1O theory,9ff. union, 10, 13 Sigma algebra, 22 Significance level, 407 Significance testing, 407, 409 Simple hypothesis (see Hypotheses, statistical) Singular continuous c. d. f., 63 Size of critical region, 407 Size of set, 29 Size of test, 407 Skewness, 75, 76 Space, 9 even~ 1, 14, 15,18,23,53 parameter, 273 probability, 25 sample, 1, 14,31 Spearman's rank correlation, 525, 526 Standard deviation, 68 Statistic, 226 Statistical hypotheses (see Hypotheses, statistical) Statistical inference, 271 Statistical tests (see Tests of hypotheses) Stieltjes integral, 69 Stirling's formula, 530 Stochastic independence, 150 Student's t distribution, 249, 250,542,550 table of, 556 Subset, 10 Sufficient statistics, 299, 301, 306, 321,391 complete, 321, 324, 326 factorization criterion, 307 jointly, 306, 307 minimal, 311, 312, 326 tests of hypotheses, 408 Summation notation, 527 Sums of random variables: covariance of, 179 distribution, 192 variance of, 178 Symmetrically distributed, 170

t distribution (see Student's t distribution) Tables of distributions, 538-543 chi-square, 553 F, 554, 555 normal, 551, 552 Student's t, 556 Target populations, 222 Taylor series, 533 Tests of hypotheses, 271,401-403 Bayes, 417, 418 chi-square, 440 composite, 418 and confidence intervals, 461 critical function of, 404 critical region of, 403 distribution-free (see Nonparametric methods) equality-of-means, 432,435 equality of two distributions, 518 equality of two multinomials, 448 goodness-of-fit, 442, 447 homogeneity (see Homogeneity of populations) homogeneity of variances, 438, 439 independence in contingency tables, 452 large-sample, 440 likelihood-ratio: generalized, 419 simple, 410,419 mean of normal population, 428-431 median, 521 minimax, 416 most powerful, 410, 411 nonrandomized, 403 null hypothesis, 405 power of, 406 randomized, 403, 404 rank-sum, 522 ratio of variances, 438 relation to confidence intervals, 461 run, 519 sequential (see Sequential tests) of significance, 407 simple, 409 size of, 407 sufficient statistics, 408 unbiased, 425 uniformly most powerful, 421 on variances, 431, 432, 438 Tests of significance, 407 Ticktacktoe problem, 62 Tolerance limits, 505, 515, 516 Total probabilities, theorem of, 35, 148, 149 Transformations, 175, 198,202 c.d.f. technique, 181 m.g.f. technique, 189 probability integral, 202, 203 Treatment effect, 437, 519

564

INDEX

Truncated distribution, 103, 104, 122, 123 normal,124 Poisson, 104 Type I and II errors, 405 size of, 405

UMVUE (uniformly minimum-variance unbiased estimator), 315 Unbiased estimator(s), 293,315,352 best linear, 499 joint, 352 uniformly minimum variance, 315 Unbiased test, 425 Uncorrelated random variables, 161, 173, 174 Uniform distribution. 105,238,540 Uniformly most accurate, 464 Uniformly most powerful test, 421 composite, 421 simple, 411 unbiased, 425 Union of sets. 10, 13

Variance, 67 analysis of, 437 conditional, 159 definition of, 67 distribution of sample, 241 estimator of, 229 of linear combination of random variables, 178,179 lower bound for, 315 sample, 229, 245 of sample mean, 231 of sum of random variables, 178 tests of, 431, 438 Variance-covariance matrix, 352, 489 Variance ratio, 246, 437 Vector parameters, 351 Venn diagrams. 11-13 Waiting time, 101, 103 Wald'sequation,470 Weibull distribution, 117, 542 Wilks' generalized variance, 353