Scribd Logo
Загрузить

Добро пожаловать в Scribd!

  • HW Problems

    Загружено:

    qbomb
    0 просмотров6 страниц
    blah

    Оригинальное название

    Hw Problems

    Авторское право

    © © All Rights Reserved

    Доступные форматы

    PDF или читайте онлайн в Scribd

    Этот документ был вам полезен?

    Это неприемлемый материал?

    Пожаловаться на этот документ
    blah

    Авторское право:

    © All Rights Reserved
    Скачайте в формате PDF или читайте онлайн в Scribd
    Отметить как неприемлемый контент
    0 просмотров6 страниц

    HW Problems

    Загружено:

    qbomb
    blah

    Авторское право:

    © All Rights Reserved
    Скачайте в формате PDF или читайте онлайн в Scribd
    Отметить как неприемлемый контент
    из 6
    “ENTROPY, RELATIVE ENTROPY AND MUTUAL INFORMATION Data inequality: If X-+¥-+Z forms @ Markov chain, then 5 Y)2 18,2) second law of thermodynamics: For a Markov chain, 1. Relative entropy D( si, 4:) decreases with time. 4» Rulative entropy Dips|iz) between a distribution and the sttionsry distribution decreases with time. 43, Entropy HUX,) inerenses ifthe stationary distribution ig uniform. ) increases with time for a stationary ‘Markov chain. 5, The conditional entropy HO%,|X,) of the initial condition Xo increases for any Markov chain. Sufficient statistic: TIX) is sufficient relative t0 (2) if and, only if. Ki, X)~ 10; TOD) for all distributions on @ Fano’s inequality: Let P,=Pr(gW)#X), where g i any funetion of Y. ‘Then AUP.) + P, log (|| = HOXY). (2.164) PROBLEMS FOR CHAPTER 2 1. Coin flips. A fair coin i flipped until the first head occurs. Let X denote the number of flips required. fo Find the entropy H(X) in bits. The following expressions may be ‘useful: (by A random variable Xn drawn according to tis datriputlon ing fan “efficent” sequence of yes-no ‘of the form, “Is X actin dhe aot 8? Compare HX) tothe expected number of {questions required to determine X. a. Entropy of functions. et X be a random variable taking on 8 Soi Entropy of ites: What is the (general) inequality relationship of HC) and HOP) if (@) ¥=2"7 (b) Y= cosX? 4. Minimum entropy. What se the minimam value of HPs.» Bod ‘HHipy as p ranges over the set of n-dimensional probabiity vootors? | Find all pis which achieve this minimam. | ‘kc Axiomatic definition of entropy. IF we assume certain exoms (96 Anocsure of information, then we will be foreed to use ‘measure like entropy. Shannon used this to justify his initial defini- tion of entropy. In this book, we will rely more on the other properties of entropy rather than its axiomatic derivation to justify its use. The following problem is considerably more difficult than the other prob- Jems in this section. Ifa sequence of symmetric functions H,(P;, Pay--+» Pm) satisfies the following propertie ‘+ Normalization: H, (3, 4)= 1, + Continuity: H,(p, 1 — p) is @ continuous function of p, + Grouping: Fy Pay +s Pe) = Hoyas + Pay Pas ++ Pu) + (Pit Pd) Hath RR) -y distribution is uniform. tses with time for a stationary «initial condition X, increases that H,, must be of the form ative to (fie) if and only if prove 3 pte p m= 2,8,-- am (2.165) san of ¥ HP Pavve-s Pn) where g is any functior . ‘There are various other axiomatic formulations which also result in ‘he samo definition of entropy. See, for example, the book by Caiszér and Korner (83) 5S. Entropy of functions of a random variable. Let X be a discrete random ‘variable. Show that the entropy of a function of X is less than or equal to the entropy of X by justifying the following ateps: Y AND MUTUAL INFORMATION PROBLEMS FOR CHAPTER 2 “3 | | ! =H) aes) HS, gO) = HX) + Hg) (2.166) 2H, (267) HOR, OR) Hg) + HEC) (2.168) sording to ‘hie distribution oe Sex) (2.169) © questions : "BIO tthe expected mami of ‘Thus H(g00)= HOO. 6. Zero conditional entropy. Show that if H(Y|X)= 0, then ¥ isa function xdom variable taking on Gnite of X, ie., for all x with p(x)>0, there is only one possible value of y reral) inequality relationship of with ptx, »)>0. 7. Pure randomness and bent coins. Let Xy,Xqy.-.+Xy denote the out- comes of independent fips of a beat com. Tous Pr(X,=1) =p, Pr(X,-0)=1~p, where p is unknown. We wish to obiain a se. imum value of HP,» P= quence Z,, Za)... Zax of fair coin fips from X,,X,,...,X,. Toward “jimensional probability vectors? this end let 7:2" {0,1}* (where {0,1}*= (A, 0, 1, 00, 61,...} is ‘imum. the set of all finite Iength binary sequences) be ‘a mapping IX Kayo Xy) = (yy ayo Zig Where Z, ~ Bernoulli (2), and K ‘certain axioms for our reese may depend on (X,, ..»%,)- In order that the sequence Z,, Z,, ‘ill be forced to use a logarithmic Elem 10. Conditional mutual information es 11. Average entropy. Let H(p)= ~P los P ENTIROPY, RELATIVE ENTROPY AND MUTUAL INFORMATION appear: the map f from bent coin flips to fair fips ‘must have the property that ‘all 2* sequences (Z,, Za »%,) of a given length k have ‘equal probability (possibly 0), for 2, geen Lene orn 2 the map (OL) 0,20) = 2, FOO) = 0) For eng, bas the property that Pr(Z = 3K =1) = Pr(Za~ to be fair coin flips, Hl p)2 BO Xe) PH, Bae Fe®) SUD + Hy.) HUD + ER) SEK. thus no more than nil(p) fair enin oases ean be derived from = X,), on the average. (7) Baht a good map f on sequences of length 4 ‘World Series. The World Serios is a seven-game series that terminaice World Series yer tear wins four games, LetX be the random varie that represents the outcome B; possible values of X are AAA, BABABAB, and BBBAAAA. Let ¥ a Pr eaber of games played, which ranges from 4 to 7. Asmumin be te nad B are equally matched and that the games are indepen Gent, caleulate HOO, HOY), HO(X), and HOX|¥). Infinite entropy. Tis problem shows that the entropy of & if evar can be infinite. Let A=2,-p(nlog! n) ° (tis eos Tao Mat A ie finite by bounding the infinite sum by the integral of ena) Show thatthe integer-valued random variable defined by PX = n)= (An log? n) farm = 2,8... has HOO = +s unconditional mutual information eee tmples of joint random variables X, ¥ and Z such that (a) WG Y|Z)< 10 Y, (by 1X; ¥12)> 106, ¥). — (a= p)log,(1 ~ p) be the bi nary entropy function. Up Rvaluate H1(1/4) using the fact that log , 9=-3.584, Hint: Gow, Evaluate FU aiment with four equally Likely outcomes, one of Which is more interesting than the others. (by Caleuate the average entropy H(p) when the probability P= ‘chosen uniformly in the range 0= p <1. TROPY AND MUTUALINFORMATION, =| PROBLEMS FOR CHAPTER 2 s tes bet) oe (© Optional) Calculate the average entropy H(p,, Py, py) where guences (2 Zaps ov) of 8 mara ts ep — lity (possibly 0), for & = 1,: . ine ueiform! ly distrib: et, 10) =1, A00)= fl) =A 1 dimension lity vector. Gener ‘that Pr(Z, = 1K=1) = Pr{Z, 12. Venn diagrams. Using Venn diagrams, we can see that the mutual ‘information common to three random variables X, ¥ and Z should be yualities: defined by 1X) WK; ¥;Z)= WX; Y)— 10; ¥1z) This quantity is symmetric in X, Y and Z, despite the preceding eK) nennogatna, ree Unortunaely, 10% 2) is not necessarily onnegative. Find at nd prove Wy Ea following two identities 1ORF:2)<0, and prov the 4K) TAX: ¥; Z)= HUY, Y, Z)— HOD) — HY) ~ H@) + 1 +1Zx) IXY; Z)= HOY, ¥, Z)~ HK, ¥) — HUY, Z)— Hz, X) + HOC) + HWY) + Hz) V+ 1%; Z) din tosses can be derived from (X,, onces of length 4 The first identi identity ean be understood using the Venn diagram anal severe sero tit erminaee ReagnerY and mutual infomation. The secind identity fll orld Series between teams A and * BABABAB, and BBBAAAA. Let Y B ‘hich ranges from 4 to 7. Assuming esuming may not be one counterfeit coin. If there is a counterfeit cots, a ae IRR herria ee ar a weighed by a balance se that the entropy of «rete (©) Find an upper bound onthe numberof coins nso that weighs areata ice Yin He contrat in Way) and cry dade as ag the infinite sum | 1¢ integral or . valued random variable X defined (©) Difficult) What is the coin weighing strategy for k = 3 weigh eos... has HX) = +2. and 12 coins? . = = s.amcondtonal mua infomation. 1. Drawing with and without replacement ‘Aa rn contains r reds w wh site Taber sea a i hse of p ~ (1 ~ p)10g,(1.~p) be the bi- that log , 31.584, Hint: Con- 15. A metric. A function plz, y) is a metre if for all x,y, et that log, 3=1.584, * pte, YEO so eaually Tiely ontomes, one of + obs, »)= ply.) a ra in proba pS * pte, 9)=0 if and only it x= y e0=psl. + els, + aLy,2)% ptr, 2). Elements al Infnematinn Thaa= upto-d informa 15.coma i pater “ samo, nsanve enor AND MUTUALIFORUATON eee (a) Show that pS, ¥)= HOY) + HOO satisficg the ro second or Shaw th properties above. If we say that X= ¥ if there ie & access and Joon function mapping X ta Y, then the third property {= blends feo satisfied, and p(X, Y) is a metric. roy (b) Verify that p(X, ¥) can also be expressed as eee &, Y)= HOO) + HUY) ~ 2069) 170) ent cand sho = HO, ¥)- 106 ¥) im) ed = 2HUX, ¥)~ HOO HO), 2.172) ere 2H, ¥)~ HOO ‘ pothesis f 16, Example of join entropy. Let pte, 9) be given by works, ¢ y Elements « gates an x oo} covered o | ad indoding: a }ot The rela Find thermod, (a) HOO, HO). © Compet (b) HRY), HISD Rede f © AY. The (@) HO)- HOW. amet (© 10Y). é: A (© Drew a Venn diagram for the quantities in (2) Ghrough G : 17. Inequality. Show In x2 1—} for x>0. > a Th Baton of « sum. Let X and ¥ be random variables thay tee, $7 Fontotiol ce em oat yy Joy «o> Jos reapectively. Let Z = 43: » inequalit To Shoo that H(Z|i)= HG?DO. Argue that if %,¥ are indepentons thee cons Show a) zHIZ) and H(X)= HZ). ‘Thus the addition of i Gependent random variables adds uncertainty. ) Gee an example (of neceasarily dependent random variables) in which H(X) > HZ) and HO)> H@). {@ Under what conditions does H(Z)~ HD+ HO? 19, Entropy of a disjoint mixture, Let X, and X, be discrete random enter ef tem necording to probability mass functions py") ane ToC) over the respective alphabets = (2, vm} B= (mth, n)- Let .X,, with probability 6» or with probability 2 — a (@) Find H(X) in terms of HU%,) and HO%,) and a. (b) Maximize over a to show. “that 2%? < 240) 4.2% and interpret Ming the notion that 2° i the effective alphabet site x => MUTUAL INFORMATION PROBLEMS FOR CHAPTER 2 NTROPY ANI . 7 ' 20. A measure of correlation. Let X, and X, be identically distributed, but HOYLX) satisfies the tee oe ‘not necessarily independent. Let f we say is ‘to Y, then the third property 21 FIX) metric. ent HK) expressed a8 eam (@) Show p= 10%; X,)/HUK,). ¥)~ 215 ¥) (b) Show 0 ps1. eam © When is p=0? 1) mispei? ne) (ear) @) When is p =17 ~ HK) ~ 2. Data processing. Let X,-> Xy-> Xy->---—»X, frm a Marko chain in we given by this order; ie, lot ) be given, Pot ay 1 t4) ples Pleals,)-- pla,la,_,), on Reduce 10%; Xy,...,,) to its simplest form. ra 22, Bottleneck. Suppose a (non- ?) Markov chain starts in one of 7 stationary) 04 7, Atates, necks down to A

    Вам также может понравиться