Instructor: |
Term: Term I (Fall) 2017 |
Dr. Steve Hranilovic |
Lectures: Thursdays 8:30-11:30am, LSB/B130E |
Office: ITB/A224 |
Email: |
Office Hours: After class or by appointment |
 |
|
This course is an introductory look into the broad areas of information
theory and coding theory founded by Claude E. Shannon in his 1948
landmark paper,
A Mathematical Theory of Communication.
As stated in the course text,
Information theory answers two fundamental questions in communication
theory: what is the ultimate data compression (answer: the entropy H) and
what is the ultimate transmission rate of communication (answer: the channel
capacity C).
|
 |
In later stages of the course, coding techniques will be discussed which
approach these ultimate limits.
| C.E. Shannon (1916-2001) (Image © IEEE) |
The course will cover as many of the following topics as time permits:
- Entropy
- entropy, relative entropy, mutual information, chain
rules, data processing inequality, the asymptotic equipartition
property, entropy rates for stochastic processes.
- Data Compression
- the source coding theorem, Kraft inequality, Shannon-Fano codes,
Huffman codes, universal source codes.
- Channel Capacity
- discrete channels, random coding bound and converse,
Gaussian channels, parallel Gaussian channels and
"water-pouring", bandlimited channels.
- Error Control Coding
- linear block codes and their properties, hard-decision decoding,
convolutional codes, Viterbi decoding algorithm,
iterative decoding.
Course text:
- T.M. Cover and J.A. Thomas, Elements of Information
Theory, John Wiley & Sons, 1991. (on reserve at Thode Library)
Additional References:
- S.B. Wicker, Error Control Systems for Digital Communication
and Storage, Prentice-Hall, 1995. (on reserve at Thode Library)
- R.G. Gallager, Information Theory and Reliable
Communication, John Wiley & Sons, Inc., 1968. (on reserve at
Thode Library)
- J.R. Pierce, An Introduction to Information Theory: Symbols,
Signals and Noise, second edition, Dover Publications Inc,
1980.
- J.M. Wozencraft and I.M. Jacobs, Principles of Communication
Engineering, John Wiley & Sons, 1965.
- Selected journal articles and papers to be cited in course.
Course Links:
Useful Links:
- C.E. Shannon, A Mathematical Theory of Communication,
Bell System Technical Journal, vol.27, pp.379-423, 623-656 July,
October 1948. (Download)
- C.E Shannon, Communication in the Presence of Noise,
Proceedings of the IRE, vol.37, no.1, pp.10-21, 1949.(reprinted in
Proceedings of the IEEE, vol.86, no.2, pp.447-457,
February 1998.
Download from McMaster Library
e-Journals site)
- David MacKay's
A Short Course in Information Theory
- Canadian Society
of Information Theory (CSIT)
- IEEE Information Theory
Society
Policy Reminders:
Senate and the Faculty of Engineering require all course outlines to
include the following reminders:
The Faculty of Engineering is concerned with ensuring an environment
that is free of all adverse discrimination. If there is a problem, that
cannot be resolved by discussion among the persons concerned, individuals
are reminded that they should contact the Department Chair, the Sexual
Harassment Officer or the Human Rights Consultant, as soon as
possible.
Students are reminded that they should read and comply with the Statement
on Academic Ethics and the Senate Resolutions on Academic Dishonesty as
found in the Senate Policy Statements distributed at registration and
available in the Senate Office.
Academic dishonesty consists of misrepresentation by deception or by other
fraudulent means and can result in serious consequences, e.g. the grade of
zero on an assignment, loss of credit with a notation on the transcript
(notation reads: "Grade of F assigned for academic dishonesty"), and/or
suspension or expulsion from the university.
It is your responsibility to understand what constitutes academic
dishonesty. For information on the various kinds of academic dishonesty
please refer to the Academic Integrity Policy, specifically Appendix 3,
located at
http://www.mcmaster.ca/senate/academic/ac_integrity.htm .
The following illustrates only three forms of academic dishonesty:
- Plagiarism, e.g. the submission of work that is not one's own or for
which other credit has been obtained.
- Improper collaboration in group work.
- Copying or using unauthorized aids in tests and examinations.
|