Course Title: IEG 5154 Information Theory

Description: The course aims to cover

1. Fundamental definitions of information measure (entropy, conditional entropy, mutual information) and their properties.
2. Lossless Source Coding/Data Compression -- theory and algorithms
3. Channel Coding/Error-Correcting Codes/Coding theory
4. Rate-distortion theory.

5. Miscellaneous advanced topics depending on time and interest (Kolmogorov Complexity, Universal Portfolio Theory, Network Coding, ...)

Topic

Contents/fundamental concepts

• Information theoretic quantities

• Lossless source coding

• Channel coding

• Rate-distortion theory

• Entropy, conditional entropy, mutual information, divergence, differential entropy, Markov/ergodic sources, properties (chain rules, positivity, convexity, Jensen's inequality, Fano's inequality, conditioning, data processing inequality, Law of large numbers, Sanov's theorem, AEP, entropy rates) 

• Achievability and converse proofs, Kraft inequality, codes -- Shannon-Fano-Elias, Huffman, Arithmetic, Lempel-Ziv (universal codes)

• Hamming codes, channel capacity (achievability and converse proofs), zero-error capacity, joint source-channel coding, feedback capacity, Gaussian channels (parallel channels, coloured noise)

• Scalar/vector quantization, rate-distortion theorem (achievability and converse)

Learning outcomes:

1. Demonstrate ability to manipulate basic information-theoretic quantities to prove relevant theorems.
2. Demonstrate understanding of foundational topics in information theory, and an ability to use the theoretic tools required to prove corresponding theorems.
3. Use the above to characterize and design information storage, manipulation and transmission systems.

Lecture

Problem Sets

Online Activities (Scribe Notes/Discussion)

Homeworks

(hr)
in class

(hr)
in/out class

(hr)
out of class

(hr)
out of class

36

0

36

12

12

6

15

0

M

O

M

O

M

O

M

O

Task nature

Description

Weight

Problem sets

Homeworks

Scribe Notes

Class participation

Final Exam 

In-class problems, handed in next class

Collaborative homeworks

Scribe notes of a particular lecture

In-class discussion/Discussion on wiki

Examination 

20% (8)

20% (4)

20% (2)

15%

25% 

Recommended Textbooks (first two available in the CUHK Bookstore, third free online):

1. Elements of Information Theory by T M Cover & J A Thomas, Wiley 2006
2. Information Theory and Network Coding by Raymond W. Yeung, Springer 2008

    3. Lecture Notes on Network Information Theory by Abbas El Gamal & Young-Han Kim, 2010

Professor/Lecturer/Instructor:

Name:

Prof. Sidharth Jaggi

Office Location:

SHB Room 706

Telephone:

2609-4326

Email:

jaggi [AT] ie {dot} cuhk {dot} edu {dot} hk

Teaching Time and Venue:

Mon 11:30am to 1:00pm, (ELB 202),

Fri 3:30pm to 5:00pm, (ERB 703).

Website:

http://ieg5154.pbwiki.com

Other information:

Office Hours: by appointment

Teaching Assistant/Tutor:

Name:

Mr. Cho Yiu Ng (Michael)

Office Location:

SHB Room 826A

Telephone:

2609-8383

Email:

michaelng [AT] ieee {dot} org 

Website:

http://ieg5154.pbwiki.com

Other information:

Office Hours: Thu 5:30pm-6:15pm

Course announcements and materials will be posted on the course wiki https://ieg5154.pbwiki.com

Attention is drawn to University policy and regulations on honesty in academic work, and to the disciplinary guidelines and procedures applicable to breaches of such policy and regulations. Details may be found at http://www.cuhk.edu.hk/policy/academichonesty/ . With each assignment, students will be required to submit a statement that they are aware of these policies, regulations, guidelines and procedures.