lecture note 3
`
Since is very small, from the right-hand side of the inequality we have
Definition(From the textbook): the relative entropy between two probability mass functions p(x) and q(x) is defined as
.(divergence), where the denotes the sample space of x.
if ||p-q|| we have
the number of typical
Question :
to prove
by l'Hôpital's rule, we have
the number of the typical type-class and the number of the typical element
the size of largest set class
Expected # bits (the typical part + the atypical part)
the typical set decrease, as the epsilon decrease.
p <1/2 and n =10 in the following figures
Fig1: the relation between #of heads and probability,
Fig2: between # of heads and size of T(k,n)
the largest value of T(k,n) = T(n/2, n) when k =n/2
Most likely of T(k,n) = T(np, n), when k=np
BSCT
(a) ,code with expected rate
(b) code with and expected rate <
For General case(i try my best to recall, but... Sorry here i just show a ref about source coding theorem)
Comments (0)
You don't have permission to comment on this page.