| 
  • If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • You already know Dokkio is an AI-powered assistant to organize & manage your digital files & messages. Very soon, Dokkio will support Outlook as well as One Drive. Check it out today!

View
 

Scribe Notes 5-2 Mar 2

Page history last edited by Silas 15 years ago

Differential Entropy

 

Prerequisite:

Let Formula be a function defined on Formula. Let Formula be a positive number and  Formula be an arbitrary number inside the interval Formula.  Define the Riemann Integral Formula as Formula  whenever the limit exists. Please refer to http://en.wikipedia.org/wiki/Riemann_integral for a more rigorous establishment of Riemann Integral.

 

Recall:

If Formula is a discrete random variable, the entropy is Formula where Formula takes values in a discrete alphabet Formula  under a probability mass function Formula.

  

Definition of differential entropy:

Now, if Formula is a continuous random variable, let Formula be the probability density function of Formula on a continuous set Formula, e.g. Formula. Let  Formula be the support of Formula. Define the differential entropy Formula

 

Two examples illustrating the properties of Formula:

 

Example 1:

Formula

 Formula 

 

Example 2:

Formula 

Formula 

 

Exercise 1:

Construct a random variable Formula with probability density function Formula such that Formula exists but Formula does not exist.

 

Suggested answer: Consider Formula for some constant Formula.

 

For any continuous random variable Formula discussed in the rest of this note, the corresponding Formula exists.

 

An interpretation of Formula: 

Suppose the support of Formula is inside the closed interval Formula. From the Prerequisite above, Formula where Formula is a positive number and  Formula is an arbitrary number inside the interval Formula. Note that Formula

 

Define the entropy of the quantized version Formula.Since Formula is a probability density function,  Formula as Formula. Therefore, it is intuitive to guess that the statement " Formula as Formula " is true. A rigorous proof of the above statement is provided in Cover's book on p.247 and 248. In their proof, the mean value theorem is used to obtain Formula for sufficiently small Formula. In addition, their proof applies to a more general case that the support of Formula can be Formula.

 

Why is Formula called "differential entropy"?

It is clear that Formula and Formula as Formula. The quantity Formula can be viewed as Formula, which is similar to the entropy of some discrete uniform random variable. Therefore, Formula can be viewed as the limit of the difference of entropies between two random variables, which may explain why Formula is called "differential entropy".

 

 

On n-bit quantization of X:

If we quantize Formula such that each quantization step has the same length Formula, the number of bits required to specify which step Formula falls into is  Formula when Formula is small. The n-bit quantization of Formula is equivalent to setting Formula for quantization. The entropy of an n-bit quantization of Formula is Formula. Therefore, Formula  is the number of bits on the average required to describe an n-bit quantization of Formula. In Example 1, The number of bits required to describe an n-bit quantization of Formula is Formula In other words, Formula bits suffice to describe Formula in Example 1 to n-bit accuracy. Similarly, the number of bits required to describe Formula in Example 2to n-bit accuracy is Formula.

 

Properties of  Formula:

1. Formula can be negative.

2. Formula may not exist.

3. Define  Formula.

4. Define Formula.

5. Define Formula.

 

Jensen's Inequality for a continuous random variable Formula :

Formula for "reasonable" convex Formula.

 

Consequences of Jensen's Inequality :

1. Formula.

2. Formula.

3. Formula.

4. Data processing inequality: If Formula, Formula.

5. Chain rule for Formula: Formula.

6. Chain rule for Formula: Formula.

 

Exercise 2 (EXTRA CREDIT):  

Show that Formula  where Formula.

 

An upper bound of Formula for Formula with Formula:

If Formula, then Formula.

Proof: Let Formula be the probability density function of a real continuous random variable. Then, Formula by Definition. Now, let Formula be the probability density function of a real normal random variable with mean Formula and variance Formula. More specifically, Formula for all Formula. We can write Formula. Note that Formula and Formula. Therefore,

Formula .

Now, Formula. Consequently, Formula.

 

The achievability of the upper bound of Formula for Formula with Formula:

Compute Formula. Since Formula,  When Formula is the random variable with probability density function Formula, Formula.

 

The entropy of a multivariate normal distribution:

The probability density function of Formula with mean Formula and convariance matrix Formula is Formula. Then, Formula. The proof is contained in Cover's book on p.250.

 

 

Comments (0)

You don't have permission to comment on this page.