Difference between differential entropy and entropy(for discrete r.v.)
1. Differential entropy can be negative;
2. In general, differential entropy has no upper bound. With given variance, Gaussian distribution maximizes the differential entropy.
3. Change of mean won't change its differential entropy, but scaling may change differential entropy.
Typical set for continuous random variables
The definition is similar to that of weakly typical set of discrete random variables.
Def: A set of sequences is said to be a typical set with respect to p.d.f f(x) if it contains all sequences s.t.
, or equivalently,
.
Some properties of typical set for continuous r.v.
1. (since )
2.
3.
To prove 2 & 3, use the definition and the following inequality:
Error estimation
For discrete r.v., we have Fano's Inequality .
However, for continuous r.v, we cannot use Fano's idea to estimate error because .
We use mean square instead.
Two facts: I. where denotes the mean of X
II. The differential entropy for is
Then, //mean of X gives the best estimation
//Gaussian distribution maximizes the differential entropy
for given variance. The unit of h(X) here is bits.
"=" iff X is Gaussian and X-hat is equal to E(X).
Therefore,
Channel Coding(PS6 Q1)
Channel capacity
(a) (Converse)
When X is uniformly distributed, H(X)=1 and we get I(X;Y) is exactly equal to 1. (Achievability)
Then the channel capacity=1
(b) Similar to (a), channel capacity=1
(e) Calculate I(X;Y) by its definition, we get I(X;Y)=(1-p)H(X). Therefore, channel capacity=1-p.
Another example,
Channel: p(0|0)=p(1|0)=p(1|1)=p(2|1)=p(2|2)=p(3|2)=p(3|3)=p(1|3)=0.5
I(X;Y)=H(Y)-H(Y|X)=H(Y)-1 To maximize I(X;Y) is equivalent to maximize H(Y).
When X is uniformly distributed, H(Y)=2 is maximized. Thus, channel capacity=1
Comments (2)
Silas said
at 5:01 pm on Mar 9, 2009
It is not clear to me why the last sentence of the section of Error Estimation holds. Can someone help me with a rigorous proof?
Cho Yiu Ng said
at 8:18 pm on Mar 10, 2009
Can anyone modify the proof to give that result?
You don't have permission to comment on this page.