Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd Logo
Upload

Welcome to Scribd!

  • 18 views

    University

    Uploaded by

    Sudo27
    This document contains student Nguyen Dinh Hoang's answers to two questions on Information Theory Assignment 3. Question 1 examines the mutual information between different sides of coins and dice. The student calculates that the mutual information is log2(2) for a coin and log2(6) for a standard 6-sided die between the top and bottom sides, and log2(3)-1 between the top side and front face of a die. Question 2 considers the entropy H(X) of the number of coin flips X needed to get the first head. The student calculates H(X) to be 2 bits.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    University

    Uploaded by

    Sudo27
    18 views2 pages
    This document contains student Nguyen Dinh Hoang's answers to two questions on Information Theory Assignment 3. Question 1 examines the mutual information between different sides of coins and dice. The student calculates that the mutual information is log2(2) for a coin and log2(6) for a standard 6-sided die between the top and bottom sides, and log2(3)-1 between the top side and front face of a die. Question 2 considers the entropy H(X) of the number of coin flips X needed to get the first head. The student calculates H(X) to be 2 bits.

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    This document contains student Nguyen Dinh Hoang's answers to two questions on Information Theory Assignment 3. Question 1 examines the mutual information between different sides of coins and dice. The student calculates that the mutual information is log2(2) for a coin and log2(6) for a standard 6-sided die between the top and bottom sides, and log2(3)-1 between the top side and front face of a die. Question 2 considers the entropy H(X) of the number of coin flips X needed to get the first head. The student calculates H(X) to be 2 bits.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    18 views2 pages

    University

    Uploaded by

    Sudo27
    This document contains student Nguyen Dinh Hoang's answers to two questions on Information Theory Assignment 3. Question 1 examines the mutual information between different sides of coins and dice. The student calculates that the mutual information is log2(2) for a coin and log2(6) for a standard 6-sided die between the top and bottom sides, and log2(3)-1 between the top side and front face of a die. Question 2 considers the entropy H(X) of the number of coin flips X needed to get the first head. The student calculates H(X) to be 2 bits.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 2

    Student Name: Nguyen Dinh Hoang

    Student ID: 20021361

    Information Theory (INT2044E)


    Assignment 3

    Question 1 Problem 4
    Mutual information of heads and tails.

    (a) Consider a fair coin flip. What is the mutual information between the top side and the
    bottom side of the coin?

    (b) A 6-sided fair dice is rolled. What is the mutual information between the top side and the
    bottom side?

    (c) What is the mutual information between the top side and the front face (the face most
    facing you)?

    (a) We have the chain rule: H(B) − H(B|T ) = I(T ; B) = log2 (2)

    (b) Same with coin flip: H(B) − H(B|T ) = I(T ; B) = log2 (2)

    (c) We use X represent for Top and Y for Front:

    H(X) = log2 (6)


    If the top is given, so the front will have four equally likely value:

    H(X|Y = 1) = log2 (4)

    and H(X|Y ) is the average of the H(X|Y = y) values; since they are all equally likely

    H(X|Y ) = log2 (4) = 2

    Lastly,
    I(X; Y ) = H(X) − H(X|Y ) = log2 (6) − 2 = log2 (3) − 1

    Question 2 Problem 7
    Coin flip. A fair coin is flipped until the first head occurs. Let X denote the number of flips
    required.
    (a) Find the entropy H(X) in bits. The following expressions may be useful:
    ∞ ∞
    X 1 X r
    rn = , nrn =
    n=0
    1 − r n=0 (1 − r)2

    1
    If X = n that means Tail occurs n − 1 times and the last flip is Head.
     n−1
    1 1
    P (X = n) = ×
    2 2
    And then,
    ∞  n  n
    X 1 1
    H(X) = − log2 ( )
    n=1
    2 2
    ∞ ∞
    −n
    n2−n
    X X
    = (2 log2 (2n ) =
    n=1 n=1
    ∞  n 1
    X 1 2
    = n = = 2.
    n=1
    2 (1 − 12 )2

    You might also like