Department of Physics

10ΕΛΕ21 Information Theory and Coding

Course Webpage:  https://eclass.uoa.gr/modules/course_info/index.php?course=DI576

Course outline

Course Content

• C.E. Shannon: His life, work and influence on modern communications.
• Information measures and basic properties: Entropy, reciprocal information, KL divergence, convexity.
• Typicality and asymptotic equipartition property.
• Stationary (ergodic ) sources and entropy rate.
• Lossless source compression. Prefix codes. Fundamental compression limits based on the entropy rate. Shannon codes. Hufman codes.
• Channel capacity. Examples (binary symmetric channel, binary erasure channel) and properties. Formulation and proof of the channel coding theorem for discrete memoryless channels. Feasibility. Joint typicality.
• Fano inequality and converse to the coding theorem. Feedback capacity.
• Continuous-time sources and channels. Differential entropy. Reciprocal information and properties. Entropy of a normal random vector.
• The additive gaussian channel. Typicality, coding theorem. ΑWGN channel capacity. Bandlimited channel capacity.
• Parallel gaussian channels. Channel with colored noise. Power allocation for rate maximization. Water-filling method.
• Sequential source coding. Arithmetic coding and Lempel-Ziv coding.
• Introduction to rate-distortion theory and lossy compression.
• Linear codes. Description and coding. Hamming codes.
• Convolutional trellis codes. Viterbi algorithm and decoding.
• Turbo codes. Iterative decoding and BCJR algorithm.
• LDPC codes. Factor graphs and Tanner graphs. Decoding with the message-passing algorithm.