Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
37 views

Lecture Notes 1 36-705 Brief Review of Basic Probability

This document provides a summary of basic probability concepts. It begins with an overview of random variables and their distributions, including cumulative distribution functions, probability mass functions, and probability density functions. It then discusses expected values, transformations of random variables, independence, and important probability distributions such as the normal, binomial, and Poisson distributions. The document concludes with sections on the sample mean, sample variance, and their sampling distributions.

Uploaded by

Pranav Singh
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
37 views

Lecture Notes 1 36-705 Brief Review of Basic Probability

This document provides a summary of basic probability concepts. It begins with an overview of random variables and their distributions, including cumulative distribution functions, probability mass functions, and probability density functions. It then discusses expected values, transformations of random variables, independence, and important probability distributions such as the normal, binomial, and Poisson distributions. The document concludes with sections on the sample mean, sample variance, and their sampling distributions.

Uploaded by

Pranav Singh
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 7

Lecture Notes 1

36-705
Brief Review of Basic Probability
I assume you already know basic probability. Chapters 1-3 are a review. I will assume
you have read and understood Chapters 1-3. If not, you should be in 36-700.

Random Variables

Let be a sample space (a set of possible outcomes) with a probability distribution (also
called a probability measure) P . A random variable is a map X : R. We write
P (X A) = P ({ : X() A})
and we write X P to mean that X has distribution P . The cumulative distribution
function (cdf ) of X is
FX (x) = F (x) = P (X x).
A cdf has three properties:
1. F is right-continuous. At each x, F (x) = limn F (yn ) = F (x) for any sequence
yn x with yn > x.
2. F is non-decreasing. If x < y then F (x) F (y).
3. F is normalized. limx F (x) = 0 and limx F (x) = 1.
Conversely, any F satisfying these three properties is a cdf for some random variable.
If X is discrete, its probability mass function (pmf ) is
pX (x) = p(x) = P (X = x).
If X is continuous, then its probability density function function (pdf ) satisfies
Z
Z
p(x)dx
pX (x)dx =
P (X A) =
A

and pX (x) = p(x) = F 0 (x). The following are all equivalent:


X P,

X F,

X p.

Suppose that X P and Y Q. We say that X and Y have the same distribution
if P (X A) = Q(Y A) for all A. In that case we say that X and Y are equal in
d
distribution and we write X = Y .

Lemma 1 X = Y if and only if FX (t) = FY (t) for all t.


1

Expected Values

The mean or expected value of g(X) is


Z
E (g(X)) =

Z
g(x)dF (x) =

( R
g(x)p(x)dx if X is continuous

g(x)dP (x) = P g(xj )p(xj )


if X is discrete.
j

Recall that:
P
P
1. E( kj=1 cj gj (X)) = kj=1 cj E(gj (X)).
2. If X1 , . . . , Xn are independent then
E

n
Y

!
Xi

i=1

E (Xi ) .

3. We often write = E(X).


4. 2 = Var (X) = E ((X )2 ) is the Variance.
5. Var (X) = E (X 2 ) 2 .
6. If X1 , . . . , Xn are independent then
Var

n
X

!
ai X i

i=1

a2i Var (Xi ) .

7. The covariance is
Cov(X, Y ) = E((X x )(Y y )) = E(XY ) X Y
and the correlation is (X, Y ) = Cov(X, Y )/x y . Recall that 1 (X, Y ) 1.

The conditional expectation of Y given X is the random variable E(Y |X) whose
value, when X = x is
Z
E(Y |X = x) = y p(y|x)dy
where p(y|x) = p(x, y)/p(x).

The Law of Total Expectation or Law of Iterated Expectation:


Z


E(Y ) = E E(Y |X) = E(Y |X = x)pX (x)dx.
The Law of Total Variance is




Var(Y ) = Var E(Y |X) + E Var(Y |X) .
The moment generating function (mgf ) is

MX (t) = E etX .
d

If MX (t) = MY (t) for all t in an interval around 0 then X = Y .

(n)

Exercise (potential test question): show that MX (t)|t=0 = E (X n ) .

Transformations

Let Y = g(X) where g : R R. Then


Z
FY (y) = P(Y y) = P(g(X) y) =

pX (x)dx
A(y)

where
Ay = {x : g(x) y}.
The density is pY (y) = FY0 (y). If g is monotonic, then


dh(y)

pY (y) = pX (h(y))
dy
where h = g 1 .
Example 2 Let pX (x) = ex for x > 0. Hence FX (x) = 1 ex . Let Y = g(X) = log X.
Then
FY (y) = P (Y y) = P (log(X) y)
y
= P (X ey ) = FX (ey ) = 1 ee
y

and pY (y) = ey ee for y R.


Example 3 Practice problem. Let X be uniform on (1, 2) and let Y = X 2 . Find the
density of Y .
3

Let Z = g(X, Y ). For example, Z = X + Y or Z = X/Y . Then we find the pdf of Z as


follows:
1. For each z, find the set Az = {(x, y) : g(x, y) z}.
2. Find the CDF
Z Z
FZ (z) = P (Z z) = P (g(X, Y ) z) = P ({(x, y) : g(x, y) z}) =

pX,Y (x, y)dxdy.


Az

3. The pdf is pZ (z) = FZ0 (z).


Example 4 Practice problem. Let (X, Y ) be uniform on the unit square. Let Z = X/Y .
Find the density of Z.

Independence

X and Y are independent if and only if


P(X A, Y B) = P(X A)P(Y B)
for all A and B.
Theorem 5 Let (X, Y ) be a bivariate random vector with pX,Y (x, y). X and Y are independent iff pX,Y (x, y) = pX (x)pY (y).
X1 , . . . , Xn are independent if and only if
P(X1 A1 , . . . , Xn An ) =

n
Y

P(Xi Ai ).

i=1

Qn

Thus, pX1 ,...,Xn (x1 , . . . , xn ) = i=1 pXi (xi ).


If X1 , . . . , Xn are independent and identically distributed we say they are iid (or that they
are a random sample) and we write
X1 , . . . , X n P

or

X1 , . . . , X n F

or

Important Distributions

Normal (Gaussian). X N (, 2 ) if
1
2
2
p(x) = e(x) /(2 ) .
2
4

X1 , . . . , Xn p.

If X Rd then X N (, ) if


1
T 1
exp (x ) (x ) .
p(x) =
(2)d/2 ||
2
P
Chi-squared. X 2p if X = pj=1 Zj2 where Z1 , . . . , Zp N (0, 1).
1

Bernoulli. X Bernoulli() if P(X = 1) = and P(X = 0) = 1 and hence


p(x) = x (1 )1x

x = 0, 1.

Binomial. X Binomial() if
 
n x
p(x) = P(X = x) =
(1 )nx
x

x {0, . . . , n}.

Uniform. X Uniform(0, ) if p(x) = I(0 x )/.


x

Poisson. X Poisson() if P (X = x) = e x! x = 0, 1, 2, . . .. The E (X) = Var (X) =


t
and MX (t) = e(e 1) . We can use the mgf to show: if X1 Poisson(1 ), X2 Poisson(2 ),
independent then Y = X1 + X2 Poisson(1 + 2 ).
Multinomial. The multivariate version of a Binomial is called a Multinomial. Consider
drawing a ball from an urn with has balls with kPdifferent colors labeled color 1, color
2, . . . , color k. Let p = (p1 , p2 , . . . , pk ) where
j pj = 1 and pj is the probability of
drawing color j. Draw n balls from the urn (independently and with replacement) and let
X = (X1 , X2 , . . . , Xk ) be the count of the number of balls of each color drawn. We say that
X has a Multinomial (n, p) distribution. The pdf is


n
p(x) =
px1 1 . . . pxkk .
x1 , . . . , x k
Exponential. X exp() if pX (x) = 1 ex/ , x > 0. Note that exp() = (1, ).
Gamma. X (, ) if
pX (x) =
for x > 0 where () =

R
0

1
x1 ex/
()

1 1 x/
x e
dx.

Remark: In all of the above, make sure you understand the distinction between random
variables and parameters.

More on the Multivariate Normal. Let Y Rd . Then Y N (, ) if




1
1
T 1
p(y) =
exp (y ) (y ) .
(2)d/2 ||1/2
2
5

Then E(Y ) = and cov(Y ) = . The moment generating function is




tT t
T
M (t) = exp t +
.
2
Theorem 6 (a). If Y N (, ), then E(Y ) = , cov(Y ) = .
(b). If Y N (, ) and c is a scalar, then cY N (c, c2 ).
(c). Let Y N (, ). If A is p n and b is p 1, then AY + b N (A + b, AAT ).
Theorem 7 Suppose that Y N (, ). Let
 
 


Y1
1
11 12
Y =
, =
, =
.
Y2
2
21 22
where Y1 and 1 are p 1, and 11 is p p.
(a). Y1 Np (1 , 11 ), Y2 Nnp (2 , 22 ).
(b). Y1 and Y2 are independent if and only if 12 = 0.
(c). If 22 > 0, then the condition distribution of Y1 given Y2 is
1
Y1 |Y2 Np (1 + 12 1
22 (Y2 2 ), 11 12 22 21 ).

(1)

Lemma 8 Let Y N (, 2 I), where Y T = (Y1 , . . . , Yn ), T = (1 , . . . , n ) and 2 > 0 is a


scalar. Then the Yi are independent, Yi N1 (, 2 ) and
 T 
Y TY

||Y ||2
2
=
n
.
2
2

2
Theorem 9 Let Y N (, ). Then:
(a). Y T 1 Y 2n (T 1 ).
(b). (Y )T 1 (Y ) 2n (0).

Sample Mean and Variance

Let X1 , . . . , Xn P . The sample mean is


Xn =

1X
Xi
n i

and the sample variance is


Sn2 =

1 X
(Xi X)2 .
n1 i

The sampling distribution of X n is


Gn (t) = P(X n t).
Practice Problem. Let X1 , . . . , Xn be iid with = E(Xi ) = and 2 = Var(Xi ) = 2 .
Then
2
E(X n ) = , Var(X n ) = , E(S 2 ) = 2 .
n
6

Theorem 10 If X1 , . . . , Xn N (, 2 ) then
2

(a) X n N (, n ).
(b)

2
(n1)Sn
2

2n1 .

(c) X n and Sn2 are independent.

You might also like