Lecture 1 (01/12/2026) (Syllabus, Lecture Slides)

• Syllabus
• Introduction
• Lecture video: here.

Lecture 2 (01/14/2026) (Lecture Notes, Lecture Review Slides)

• What is information?
• How do we measure information?
• Lecture video: here. Old video: here.

Lecture 3 (01/16/2026) (Lecture Notes, Lecture Review Slides)

• How do we measure information (entropy)?
• Lecture video: here.

Lecture 4 (01/21/2026) (Lecture Notes, Lecture Review Slides)

• Example of entropy, self information and properties of entropy
• Joint self information and joint entropy
• Conditional self information and conditional entropy
• Lecture video: here.

Lecture 5 (01/23/2026) (Lecture Notes, Lecture Review Slides)

• Chain rule
• Mutual information
• Lecture video: here.

Lecture 6 (01/26/2026) (Lecture Notes, Lecture Review Slides)

• Conditional mutual information
• Relative entropy
• Lecture video: here.

Lecture 7 (01/28/2026) (Lecture Notes, Lecture Review Slides)

• Relative entropy
• More on chain rule and more on mutual information
• Lecture video: here.

Lecture 8 (01/30/2026) (Lecture Notes, Lecture Review Slides)

• More on chain rule and more on mutual information
• Four important inequalities
  • Jensen’s inequality
• Lecture video: here.

Lecture 9 (02/02/2026) (Lecture Notes, Lecture Review Slides)

• Four important inequalities
  • Jensen’s inequality
  • Log sum inequality
• Lecture video: Here.
• Homework 1: 2.1-2.7, 2.9, 2.12, 2.14-2.16, 2.19, 2.20, 2.25, 2.29. Solution: here. Solution video: here.

Lecture 10 (02/04/2026) (Lecture Notes, Lecture Review Slides)

• Four important inequalities
  • Log sum inequality
  • Data processing inequality
• Lecture video: Here.

Lecture 11 (02/06/2026) (Lecture Notes, Lecture Review Slides)

• Four important inequalities
  • Data processing inequality
  • Fano’s inequality
• Information Source and Asymptotic Equipartition Property (AEP)
• Lecture video: Here.

Lecture 12 (02/09/2026) (Lecture Notes, Lecture Review Slides)

• Asymptotic Equipartition Property (AEP)
• Lecture video: here.

Lecture 13 (02/11/2026) (Lecture Notes, Lecture Review Slides)

• Asymptotic Equipartition Property (AEP)
• Data compression (source coding)
• Lecture video: here.

Lecture 14 (02/13/2026) (Lecture Notes, Lecture Review Slides)

• Data compression (source coding)
• Source with memory
• Lecture video: here.

Lecture 15 (02/16/2026) (Lecture Notes, Lecture Review Slides)

• Source with memory
• Entropy rate

• Lecture video: here.

Lecture 16 (02/18/2026) (Lecture Notes, Lecture Review Slides)

• Entropy rate
• Hidden Markov Processes

• Lecture video: here.

Lecture 17 (02/20/2026) (Lecture Notes, Lecture Review Slides)

• AEP for stationary ergodic processes
• Lossless source code
  • Introduction
  • Fixed-to-variable length code

• Lecture video: here.

Lecture 18 (02/23/2026) (Lecture Notes, Lecture Review Slides)

• Lossless source code
  • Fixed-to-variable length code
  • Uniquely decodable and prefix code
  • Kraft inequality

• Lecture video: here.

Lecture 19 (02/25/2026) (Lecture Notes, Lecture Review Slides)

• Lossless source code
  • Kraft inequality
  • Optimal Codes

• Lecture video: here.
• Homework 2: 3.2, 3.9, 4.1, 4.3, 4.6, 4.7, 4.9, 4.11, 4.33, 5.1, 5.4, 5.6, 5.8, 5.12, 5.16, 5.24, 5.25, 5.30, 5.39. Solution: here. Solution video: part 1, part 2.

Lecture 20 (02/27/2026) (Lecture Notes, Lecture Review Slides)

• Lossless source code
  • Shannon codes
  • Huffman codes

• Lecture video: here.

Lecture 21 (03/02/2026) (Lecture Notes, Lecture Review Slides)

• Lossless source code
  • Huffman codes
  • The optimality of Huffman codes

• Lecture video: here.

Lecture 22 (03/04/2026) (Lecture Notes, Lecture Review Slides)

• Lossless source code
  • The optimality of Huffman codes
  • Shannon-Fano-Elias Coding

• Lecture video: here.

Lecture 23 (03/06/2026) (Lecture Notes, Lecture Review Slides)

• Lossless source code
  • Mismatched Shannon Codes
  • Competitive Optimality of Shannon Codes
  • Universal Compression of Binary Sequences (Chapter 13)
  • Lempel-Ziv (LZ78) Universal Compression (Chapter 13)

• Lecture video: here.

Lecture 24 (03/09/2026) (Lecture Notes, Lecture Review Slides)

• Channel capacity
  • Channel modeling
  • Channel Capacity

• Lecture video: here.

Lecture 25 (03/11/2026) (Lecture Notes, Lecture Review Slides)

• Channel capacity
  • Channel Capacity
  • Examples

• Lecture video: here.

Lecture 26 (03/13/2026) (Lecture Notes, Lecture Review Slides)

• Channel capacity
  • Examples
  • An Optimization Theorem

• Lecture video: here.

Lecture 27 (03/23/2026) (Lecture Notes, Lecture Review Slides)

• Channel capacity
  • An Optimization Theorem (Z-channel example)
  • Joint Typical Sequences

• Lecture video: here.
• Homework 3: 7.1, 7.2, 7.3, 7.4, 7.5, 7.7, 7.8, 7.9, 7.11, 7.12, 7.20, 7.35. Solution: here. Solution video: here.

Lecture 28 (03/27/2026) (Lecture Notes, Lecture Review Slides)

• Channel capacity
  • Proof of Channel Coding Theorem

• Lecture video: here.

Lecture 29 (03/30/2026) (Lecture Notes, Lecture Review Slides)

• Channel capacity
  • Proof of Channel Coding Theorem (continuing)

• Lecture video: here.

Lecture 30 (04/01/2026) (Lecture Notes, Lecture Review Slides)

• Channel capacity
  • Proof of Channel Coding Theorem (continuing)
  • Feedback Capacity

• Lecture video: here.

Lecture 31 (04/03/2026) (Lecture Notes, Lecture Review Slides)

• Channel capacity
  • Feedback Capacity
  • Source-Channel Separation Theorem

• Lecture video: here.

Lecture 32 (04/06/2026) (Lecture Notes, Lecture Review Slides)

• Differential Entropy
• Relative Entropy of Continuous Random Variables
• Joint and Conditional Differential Entropy

• Lecture video: here.

Lecture 33 (04/08/2026) (Lecture Notes, Lecture Review Slides)

• AEP for continuous Random Variables
• Properties

• Lecture video: here.

Lecture 34 (04/10/2026) (Lecture Notes, Lecture Review Slides)

• Additive Noise
• Capacity of Additive White Gaussian Noise (AWGN) Channel

• Lecture video: here.
• Homework 4: 8.1, 8.2, 8.4, 8.5, 8.8, 8.9, 8.10, 9.1, 9.2, 9.3, 9.5, 9.6 9.7, 9.8. Solution: here. Solution video: here.

Lecture 35 (04/13/2026) (Lecture Notes, Lecture Review Slides)

• Capacity of Additive White Gaussian Noise (AWGN) Channel
• Capacity of Other Channels

• Lecture video: here.

Lecture 36 (04/15/2026) (Lecture Notes, Lecture Review Slides)

• Capacity of Other Channels
• Quantization

• Lecture video: here.

Lecture 37 (04/17/2026) (Lecture Notes, Lecture Review Slides)

• Quantization
• Rate-Distortion Problem
• Examples

• Lecture video: here.
• Homework 5: 1, 10.2, 10.4, 10.5, 10.11, 10.12, 10.19. Solution: here. Solution video: here.

Lecture 38 (04/20/2026) (Lecture Notes, Lecture Review Slides)

• Binary Source with Hamming Distortion
• Gaussian Source with Squared-Error Distortion
• The Analytical Method for Calculation of Rate-Distortion Function

• Lecture video: here.

Lecture 39 (04/22/2026) (Lecture Notes, Lecture Review Slides)

• Arimoto-Blahut Algorithm
• Separation Theorem
• Parallel Gaussian Sources

• Lecture video: here.

Lecture 40 (Bonus, not required for the final exam)

• Network Information Theory: Distributed Source Coding

• Lecture video: here.