Information-Theoretic Learning (ITL)

Leiden University, Spring Semester 2020

General Information

LecturerProf. Dr. Peter Grünwald, Leiden University, Mathematical Institute, and Centrum Wiskunde & Informatica (CWI), Amsterdam
Teaching assistantTyron Lardy
Contact: send email to: tyronlardy at live.nl.

The URL of this webpage is www.cwi.nl/~pdg/teaching/inflearn. Visit this page regularly for changes, updates, etc.

This course is on an interesting but complicated subject. It is given at the master's or advanced bachelor's level. Although the only background knowledge required is elementary probability theory, the course does require serious work by the student. The course load is 6 ECTS. Click here (studiegids) for a general course description.

Many thanks are due to Steven de Rooij (Leiden University) who prepared a significant proportion of the exercises.

Lectures and Exercise Sessions

Because of the Corona Emergency, lectures take place remotely, via Blackboard, Kaltura Captura, each Tuesday from 14.15--16.00. The lectures are immediately followed by a virtual mini-exercise session held by Tyron Lardy. Presumably, there will abe no lecture on April 14 and May 5, but due to the Corona emergency, these plans may change. The last official lecture is scheduled for May 19, and the final exam will be held on Friday, June 5th, in the morning from 10:15 to 13:15. The exam will be sent to you by email and/or be made available on blackboard. It is an Open Book/Internet Exam, so you are free to look up things on the internet. However, I will request you to subscribe to an 'honour's code' that you do not communicate with each other during the exam.

Homework Assignments

Weekly Homework: At every lecture. The assignment is made available on this webpage. Homework is obligatory and must be turned in 24 hours before the beginning of the next lecture, i.e. six days after the assignment was handed out: on Mondays, at 14:15. You can turn in your homework digitally via blackboard or by e-mailing it to Tyron (photo's of handwritten homework are o.k.). Include your name on the first page of the pdf you hand in. Include both the number of the homework set as well as your name in the name of the pdf file. For example: ITL_HW4_Lardy.pdf After the lecture, there is (approximately) 30 minutes homework session, during which the homework will be explained and discussed by Tyron. Turning in homework in time is required, see below. Homework solutions will be made available the evening or morning before the homework is discussed.

Credit

6 ECTS points.

Examination form

In order to pass the course, one must obtain a sufficient grade (5.5 or higher) on both of the following two:
  1. An open-book written examination (to be held June 5th).
  2. Homework. Each student must hand in solutions to homework assignments at the beginning of the lecture after the homework was handed out. Discussing the problems in the group is encouraged, but every participant must write down her or his answers on her or his own. The final homework grade will be determined as an average of the weekly grades.
The final grade will be determined as the average of the two grades, with the homework counting 40% and the final exam counting 60%.

Literature

We will mainly use various chapters of the following source: P. Grünwald. The Minimum Description Length Principle, MIT Press, 2007. Some additional hand-outs will be made available free of charge as we go. For the third week, this is Luckiness and Regret in Minimum Description Length Inference, by Steven de Rooij and Peter Grünwald, Handbook of the Philosophy of Science, Volume 7: Philosophy of Statistics, 2011. This paper gives an overview of the part of this course that will be concerned with the relation between statistics, machine learning and data compression, as embodied in MDL learning.

Course Schedule

Lecture contents are subject to change at any time for any reason. A more precise schedule, with links to all exercises, will be determined as we go.
  1. February 4: introduction
  2. February 11: Preparatory Probability Theory and Statistics.
  3. February 18: data compression without probability
  4. February 25: Codes and Probabilities (the most important lecture!)
  5. March 3rd:
  6. March 10th and 17th: no lectures.
  7. March 24th: Universal Coding
  8. March 31st:
  9. April 7: Simple Refined MDL, Prequential Plugin Codes
  10. April 14: General Refined MDL, Prediction with MDL, Issues with Universal Codes/MDL
  11. April 21: Excursion: Safe Testing I.
  12. April 28: Excursion: Safe Testing II.
  13. May 5th: No Lecture! (liberation day)
  14. May 12th: Maximum Entropy
  15. May 19th: MaxEnt and MDL, Overview, Wrap Up
  16. Friday June 5th, 10.15-13.15: Exam (to be made at home, hand-in via email) Example examination.