Entropy

· Information - I(p) =
-

logp (P) b = 2 : bits ; b = 3 : trits ; b = 2 : nat ; b = 10 :

Hartleys Expected info :
#(x) :
E[I(] =
[P(x) I(X) = -

>D) 109 > P(x)

H(x) =
(PX -
I(X) + P( ! X) I (ix) .
= (0 10g0 .
1 + 0
41090
. -

9)
· Relative entropy (Rollback) : -

D(p(19) =
Ep [logp(p()/4(x)]-left Joint
Entropy H(x Y) EF : -

,
= -

109b P(X , Y)] ·

H(x Y) , = max(H(x) H(Y)) ,

EACATHLYind
each element of x

y) (0g(X y) o
=2

p(x)(ogp(p(x)(4()) ·
= 2

= =
- p(x = x , y =
= x, y =
XY

· Mutual Info :
a) I(X , Y) =
H(X) -H (X)
AH :
I (XN) = H(x) + H(Y) -
H(y, x) ↑ (BIA) P(A)
(b) I(x Y) H(y) H(Y(X) P(AIB) =

P(B) P(B(A) P(A) + P(B(A) p(A)

=
-

,
↳
P(M) =
P(MIS) p(s) -
+
P(M1(S) P(IS) .
-

H (X (y) = H (x Y) (Y)

=
conditional entropy :
,
-

(1) p qP
R
(P(MIS)10gP(MIS)
-

H(M1s)
(X(Y) p(x)
) log , yj)
==

P(xi
. P(Xi
= ·
= -

+ P(M) ! S) 1092 (P1 ! S ( , ,

· ar n p (1 p) failure

i-
=
=
P(Jj) 109 pP(Yj)
.

p success
. -
=

=& EP(xi yj) 109 ) P(xi yj)

+
/Is) , ,
· S =
ar
n = trials

with PoisSion X R
CLT in mean of sample will differ dealing when dealing with summation X any is given
-

:
que not when any - x el =
· If not
↑ (R events in interval) = 2 *
-

from =60 by more than y u R!

mean
lic X
Ue e62
=

E
ar =
x =
-

P
Y -ne =

Hav ↑ ((X 60/14) -

- P(56 = 2 [GU)
yar =

6 40 :

o X =
Ag no of events
per interval SD = Inar
R takes values 0, 1 , 2
=
.
= .
. .

u =
ne (a) =
162 =
Ar

ential Bernoulli Pl
*
ancE[x]
=
↑x p(1 -

var
=
=

2xc
= 0
+ (x ; x) =
I= [EX] <)
E(Xi P x (1-p)
-

-
el =
D
OO Y = variance f x)
interval [0 , a J (ear -

Bernoullr given
case I dist ; unrown ;
CQS C2
Case 3 :- any var n
large

e ↳
-

Known ar
-
bound
-
I
with rep lacement
=> upper estimate variance
-
E =E
: "
(1 =)
-

biased
:
test
See saneel) =
df = n -
1

um of square 11 + 12 -
2
Of deviation

df

-
= n1 + 172 -

aneas=
Xd :
After Before

Regression (x x S O
> 1092 (10)
bits bits
j
-

y
from entropy in to
x ↑ hartly
..
:

sc
-

us
...
-

b =
(y j)
- Other
bits + o not
↑ nat 1092(e) bits

=
=

bits to Hartly
C.

give (e)
2
(x -
5)
My-DMX bx
A
y' a + (y j)2
=
= -

·
To find P(A) P(B/5 M)
-(A) P(A/B E)
Blanket
= . ,
,

marginization
:
.

PCE)
use

·
modes parents
eX
! B, E

B! E
↳ (B) (A) PCMIA) : P(AB E) ,
.

↑ (T) P(J(A) P(A)

.
=

: ,,
BE
+ P(J) ! A) PCIA) < P(B)
: P(J (A) P (MA) P(AB E) P(E) + P(AIB ! E) PCIE)
.

nodes children
.
:

· ,

· nodes children's parents

For normal normal distribution

problems use !

z =
E

100 small small

large ·
Polling election
:
a
e
· SU ⑧
case Case X100 X , nu Case
B

gill-case
U-gill
-Val Bernoulli D
-

%.given Comp dis UNROWN

-case Var = d = Unknown
③
var Var giv 0
1 ⑳ & ()
: - :

5 = given
-

case x E =
giv
95 % 1
:?
eas
EIN S given
A
Q = EXi
A Margin error a =
Exi :

and