Overfitting
statistical methods for machine learning
6 CFU, MSc in Computer Science
machine learning
6 CFU, MSc in Data Science for Economics

2022-23 edition
INSTRUCTOR/DOCENTE: Nicolò Cesa-Bianchi
TAs: to be announced

News

For students of the MSc in Data Science for Economics

Bibliographic references:

Lecture notes provided by the instructor

The course makes heavy use of probability and statistics. A good textbook on these topics is:

Dimitri P. Bertsekas and John N. Tsitsiklis, Introduction to Probability (2nd edition). Athena Scientific, 2008.

Some good machine learning textbooks:
Shai Shalev-Shwartz e Shai Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press, 2014.

Mehryar Mohri, Afshin Rostamizadeh e Ameet Talwalkar, Foundations of Machine Learning, MIT Press, 2012.

L. Devroye, L. Gyorfi, and G. Lugosi, A Probabilistic Theory of Pattern Recognition, Springer, 1996.

Goals

Machine learning is concerned with the design of algorithms that can predict the evolution of a phenomenon based of a set of observations. A standard tool in the development of intelligent systems, machine learning has been successfully applied to a wide range of domains, including vision, speech and language, human-computer interaction, personalized recommendations, health and medicine, autonomous navigation, and many more. The course will describe and analyze, in a rigorous statistical framework, some of the most important machine learning techniques. This will provide the student with a rich set of methodological tools for understanding the general phenomenon of learning in machines.
Note: This is a course about the theoretical foundations of machine learning and the analysis of machine learning algorithms. The focus is on understanding the mathematical principles at the basis of machine learning.

Syllabus

  1. Introduction
  2. The Nearest Neighbour algorithm
  3. Tree predictors
  4. Statistical learning
  5. Hyperparameter tuning and risk estimates
  6. Risk analysis of tree predictors
  7. Consistency and nonparametric algorithms
  8. Linear predictors
  9. Online gradient descent
  10. Kernel functions
  11. Support Vector Machines
  12. Stability bounds and risk control for SVM
  13. Boosting and ensemble methods
  14. Neural networks and deep learning

The experimental projects will be announced here

Exams

The exam consists of two parts:

  1. Writing a paper of about 10-15 pages containing either a report describing experimental results (experimental project) or a in-depth analysis of a theoretical topic (theory project). Note: the evaluation of the theory project includes an oral discussion on the report's contents.
  2. Taking a written test on all the topics covered in class.
The written test can be only taken at the regular exam sessions. The project can be turned in at any time between June 2023 and the end of May 2024. The final grade will be computed by combining the grade obtained in the project (which includes the oral discussion for the theory project) with the grade obtained in the written test.

The experimental project is typically based on implementing two or more learning algorithms (or variants of the same algorithm) from scratch. The algorithms are compared on real-world datasets. The programming language is immaterial. However, the implementation should be reasonable in terms of running time and memory footprint. If the experimental project is based on neural networks, then the student is allowed to use a toolbox (e.g., Tensorflow). The report, preferably written using LaTeX, will be evaluated according to the following criteria:

It is OK if two students submit an experimental project together as a group. Clearly, such group projects are expected to be more extensive than single-person projects (more experiments, comparison with baseline algorithms, etc).
If your solution is adapted from other sources (e.g., Kaggle), this must be clearly stated, and the report should explain the differences and compare the experimental results.

Steps to complete the experimental project:

  1. Fill out a form to choose a project
  2. Create a public repository containing both the code and the report (in pdf)
  3. Fill out a form to turn in the project
  4. In 1-2 weeks, you will receive the grade via email.

The theory project is typically (but not exclusively) focused on a topic taught in class. The report will be based on one scientific paper (provided by the instructor), and must contain the complete proof of at least a technical result, including all necessary definitions and auxiliary lemmas. The goal of the theory project is to provide an in-depth presentation of the paper's results, including its connections with the related literature. The report may be structured as follows

Note: The theory project report MUST be written in LaTeX.

Steps to complete the theory project:

  1. Send an email directly to the instructor to agree on a topic;
  2. Turn in the report via email to the instructor;
  3. After 1-2 weeks, the instructor will get in touch to agree on a date for the oral discussion.
IMPORTANT FOR ALL PROJECTS: Your report must contain the following declaration: I/We declare that this material, which I/We now submit for assessment, is entirely my/our own work and has not been taken from the work of others, save and to the extent that such work has been cited and acknowledged within the text of my/our work. I/We understand that plagiarism, collusion, and copying are grave and serious offences in the university and accept the penalties that would be imposed should I engage in plagiarism, collusion or copying. This assignment, or any part of it, has not been previously submitted by me/us or any other person for assessment on this or any other course of study.

Course calendar:

Browse the calendar pages and click on a day to find out what was covered on that day, and the link to the video in case the class was viderecorded.