McMaster University, Faculty of Engineering  
ECE 723 - Information Theory & Coding
Department of Electrical and Computer Engineering
 


Instructor: Term: Term I (Fall) 2017
Dr. Steve Hranilovic Lectures: Thursdays 8:30-11:30am, LSB/B130E
Office: ITB/A224
Email: Office Hours: After class or by appointment
This course is an introductory look into the broad areas of information theory and coding theory founded by Claude E. Shannon in his 1948 landmark paper, A Mathematical Theory of Communication. As stated in the course text,

Information theory answers two fundamental questions in communication theory: what is the ultimate data compression (answer: the entropy H) and what is the ultimate transmission rate of communication (answer: the channel capacity C).
In later stages of the course, coding techniques will be discussed which approach these ultimate limits. C.E. Shannon (1916-2001)
(Image © IEEE)

The course will cover as many of the following topics as time permits:
Entropy
entropy, relative entropy, mutual information, chain rules, data processing inequality, the asymptotic equipartition property, entropy rates for stochastic processes.
Data Compression
the source coding theorem, Kraft inequality, Shannon-Fano codes, Huffman codes, universal source codes.
Channel Capacity
discrete channels, random coding bound and converse, Gaussian channels, parallel Gaussian channels and "water-pouring", bandlimited channels.
Error Control Coding
linear block codes and their properties, hard-decision decoding, convolutional codes, Viterbi decoding algorithm, iterative decoding.
Course text:
  • T.M. Cover and J.A. Thomas, Elements of Information Theory, John Wiley & Sons, 1991. (on reserve at Thode Library)
Additional References:
  • S.B. Wicker, Error Control Systems for Digital Communication and Storage, Prentice-Hall, 1995. (on reserve at Thode Library)
  • R.G. Gallager, Information Theory and Reliable Communication, John Wiley & Sons, Inc., 1968. (on reserve at Thode Library)
  • J.R. Pierce, An Introduction to Information Theory: Symbols, Signals and Noise, second edition, Dover Publications Inc, 1980.
  • J.M. Wozencraft and I.M. Jacobs, Principles of Communication Engineering, John Wiley & Sons, 1965.
  • Selected journal articles and papers to be cited in course.
Course Links: Useful Links:
Policy Reminders:

Senate and the Faculty of Engineering require all course outlines to include the following reminders:

The Faculty of Engineering is concerned with ensuring an environment that is free of all adverse discrimination. If there is a problem, that cannot be resolved by discussion among the persons concerned, individuals are reminded that they should contact the Department Chair, the Sexual Harassment Officer or the Human Rights Consultant, as soon as possible.

Students are reminded that they should read and comply with the Statement on Academic Ethics and the Senate Resolutions on Academic Dishonesty as found in the Senate Policy Statements distributed at registration and available in the Senate Office.

Academic dishonesty consists of misrepresentation by deception or by other fraudulent means and can result in serious consequences, e.g. the grade of zero on an assignment, loss of credit with a notation on the transcript (notation reads: "Grade of F assigned for academic dishonesty"), and/or suspension or expulsion from the university.
It is your responsibility to understand what constitutes academic dishonesty. For information on the various kinds of academic dishonesty please refer to the Academic Integrity Policy, specifically Appendix 3, located at
http://www.mcmaster.ca/senate/academic/ac_integrity.htm .

The following illustrates only three forms of academic dishonesty:
  1. Plagiarism, e.g. the submission of work that is not one's own or for which other credit has been obtained.
  2. Improper collaboration in group work.
  3. Copying or using unauthorized aids in tests and examinations.