lecture note 3

` 
Since
is very small, from the right-hand side of the inequality we have

Definition(From the textbook): the relative entropy between two probability mass functions p(x) and q(x) is defined as
.(divergence)
if ||p-q||
we have
the number of typical


Question : 
to prove 
by l'Hôpital's rule, we have
the number of the typical type-class
and the number of the typical element 
the size of largest set class
Expected # bits (the typical part + the atypical part)

the typical set decrease, as the epsilon decrease.
p <1/2 and n =10 in the following figures
Fig1: the relation between #of heads and probability,
Fig2: between # of heads and size of T(k,n)
BSCT
(a)
,
code with expected rate 
(b)
code with
and expected rate < 
Comments (0)
You don't have permission to comment on this page.