Statistical Learning and Predictive Analysis

Leiden University, Autumn 2011, 2nd year master of statistical science

General Information

The course load is 4 ECTS. Click here for a general course description.

Lectures and Exercise Sessions

Credit

Examination form

1. A written open-book examination.
2. Homework Projects. We handed out two homework assignments, one about regression and one about classification (involving setting up some experiments in R, experimenting, and writing a short report about the results). Discussing the problems in the group is encouraged, but every participant must do her or his experiments and write her or his report on her or his own. The final homework grade will be determined as an average of the grades for the two assignments.

Example of examination questions

Literature

Course Schedule

1. September, 6th: Introduction (Kotlowski)
   • General introduction: statistical learning, supervised learning, regression and classification, overfitting, linear classifiers, nearest-neighbor classification, expected prediction error and Bayes-optimal prediction rule (a.k.a. "regression function").
   • Literature: Chapter 1 and parts of Chapter 2 (Sections 2.1-2.4) of "The Elements of Statistical Learning".
2. September, 13th: Regression, part I (Kotlowski).
   • Bias-variance decomposition for squared error loss. Two views on the notion of ``model'' and ``fitting''. Linear regression: least squares, interpretation of least squares as orthogonal projection, sampling properties of estimator, z score, significance test, confidence intervals.
   • Literature: Section 2.5, Chapter 3 (Sections 3.1, 3.2).
3. September, 20th: Regression Part II. (Kotlowski)
   • Gauss-Markov theorem. Model selection and overfitting: subset selection, shrinkage methods (ridge regression and lasso).
   • Literature: Section 3.2.2, 3.3.1, 3.3.2, 3.4.1, 3.4.2.
   • homework
4. September, 27th: Classification Part I (Kotlowski)
   • Problems with Least Squares for classification; Linear Discrimant Analysis (LDA), Logistic Regression, Expected Prediction Error: 0/1 vs. logarithmic. Discriminative vs. generative models.
   • Literature: Section 4.1, 4.2, 4.3 (without 4.3.1, 4.3.2, 4.3.3), 4.4 (without 4.4.3).
   • Additional literature: Andrew Y. Ng, Michael Jordan: On Discriminative vs. Generative Classifiers: A comparison of logistic regression and Naive Bayes, NIPS 2002.
5. October, 4th: Classification Part II (Kotlowski)
   • Naive Bayes classifier; Naive Bayes and Logistic Regression; Naive Bayes and spam filtering; Optimal Separating Hyperplanes; Support Vector Machines; the Kernel Trick; SVM learning as regularized hinge loss fitting.
   • Literature: Section 4.5.2, 6.6.3, 12.2, 12.3.1, 12.3.2.
6. October, 11th: Classification Part III (Kotlowski)
   • Classification and regression trees, bagging, boosting (AdaBoost), boosting as forward stagewise additive modeling.
   • Literature: Section 9.2, 8.7, 10.1, 10.2, 10.3.
7. October, 18th: Unsupervised Learning (Kotlowski)
   • Clustering: K-means.
   • Literature: Section 14.3.1 - 14.3.6, 14.3.11.
   • homework
8. October, 25th: Nonlinear Optimal Scaling Methods (Meulman)
9. November, 1st: TBA (Meulman)

Here you can find an example of examination questions