statistical methods for machine learning 6 CFU, MSc in Computer Science machine learning and statistical learning 6 CFU, MSc in Data Science for Economics 2023-24 edition INSTRUCTOR/DOCENTE: Nicolò Cesa-Bianchi TAs: Roberto Colomboni and Emmanuel Esposito

News

• A slightly revised version of the quiz list published on December 1, 2024.

• The review of the written test of September 25 will take place on October 22 at 5 PM on Teams. This is your only chance to review your exam sheet. See the course calendar below here for the link.

• Dates of exam sessions in 2025 have been added to the calendar.

• Written test of September 25, 2024: results. There will be a zoom meeting for all those who want to review their test. The date and time will be announced here.

• The experimental projects for students who attended the course in the academic year 2023-24 or earlier are now public.

• In the academic year 2023-24, the average grade for this course so far is: 27.6 (CS), 24.0 (DSE, ML module only), 28.0 (MAT), 24.3 (ERASMUS).

• In the academic year 2022-23, the average grade for this course was: 27.2 (CS), 26.2 (DSE, ML module only), 29.0 (MAT), 25.1 (ERASMUS).

• For the past edition of this course, follow this link.

For students of the MSc in Data Science for Economics

• The course Machine learning and statistical learning has two separate exams, one for the MACHINE LEARNING module (Cesa-Bianchi, 40 hours, this course) and one for the STATISTICAL LEARNING module (Salini, 40 hours).
  The written test for the MACHINE LEARNING module will be offered on dates that do not necessarily coincide with the DSE exam sessions. These dates are shown in the calendar entries below here. DSE students who want to take the written test on one of the advertised dates must fill out a form that will be sent via email a few days before the session date.

Bibliographic references:

Lecture notes provided by the instructor

The course makes heavy use of probability and statistics. A good textbook on these topics is:

Dimitri P. Bertsekas and John N. Tsitsiklis, Introduction to Probability (2nd edition). Athena Scientific, 2008.

Shai Shalev-Shwartz e Shai Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press, 2014.

Mehryar Mohri, Afshin Rostamizadeh e Ameet Talwalkar, Foundations of Machine Learning (2nd edition), MIT Press, 2018.

Goals

Machine learning is the main enabling technology of modern artificial intelligence. This course explains the statistical foundations of machine learning, describes some fundamental algorithms for supervised learning, and shows how to analyze their performance. Emphasis is on theory and principled methods as opposed to practice and heuristics.

Syllabus

1. Introduction (version of March 5, 2024)
2. The Nearest Neighbour algorithm (version of March 17, 2024)
3. Tree predictors (version of March 20, 2024)
4. Statistical learning (version of May 28, 2024)
5. Risk analysis for tree predictors (version of April 7, 2024)
6. Hyperparameter tuning and risk estimates (version of April 8, 2024)
7. Consistency and nonparametric algorithms (version of April 16, 2024)
8. Risk analysis for Nearest Neighbour (version of May 28, 2024)
9. Linear predictors (version of May 28, 2024)
10. Online gradient descent (version of May 28, 2024)
11. Kernel functions (version of May 16, 2023)
12. Support Vector Machines (version of June 6, 2024)
13. Stability and risk control for SVM (version of May 26, 2023)
14. Neural networks and deep learning (version of May 23, 2024)
15. Logistic regression and surrogate loss functions (version of June 8, 2023)
16. Boosting and ensemble methods (version of June 1, 2024)
17. Epilogue: Generative AI

Some notebooks with small experiments showing overfitting and hyperparameter tuning.

Experimental projects for students who attended the course in the academic year 2023-24 or earlier

Warning: The current list of projects is valid until May 31, 2025. This is the only deadline. Projects turned in after May 31, 2025 will be ignored

The exam consists of two parts:

1. Writing a paper of about 10-15 pages containing either a report describing experimental results (experimental project) or a in-depth analysis of a theoretical topic (theory project).
2. Taking a written test on all the topics covered in class. The test consists in a list of 12 questions taken from this list (updated from time to time), plus an extra bonus question. During the test, it is forbidden to access any material (in paper or digital format) besides the test content.

The evaluation of the theory project also includes an oral exam on the report's contents and the related topics covered in class.

The written test can be only taken at the regular exam sessions. The project can be turned in at any time between June 2024 and the end of May 2025. The final grade is a weighted average (rounded to the nearest integer) of the mark obtained in the written test (60%) and the mark obtained in the project (40%). The exam is passed if: the average is 18 or higher and both marks are 17 or higher.

The experimental project is typically based on implementing two or more learning algorithms (or variants of the same algorithm) from scratch. The algorithms are compared on real-world datasets. The programming language is immaterial. However, the implementation should be reasonable in terms of running time and memory footprint. If the experimental project is based on neural networks, then the student is allowed to use a toolbox (e.g., Keras). The report, preferably written using LaTeX, will be evaluated according to the following criteria:

• Correctness of the general methodological approach
• Reproducibility of the experiments
• Correctness of the approach used for choosing the hyperparameters
• Clarity of exposition

Steps to complete the experimental project:

1. Fill out a form to choose a project
2. Create a public repository containing both the code and the report (in pdf)
3. Fill out a form to turn in the project
4. In 1-3 weeks, you will be contacted via email to schedule the oral examination.

The theory project is typically (but not exclusively) focused on a topic taught in class. The report will be based on one scientific paper (provided by the instructor), and must contain the complete proof of at least a technical result, including all necessary definitions and auxiliary lemmas. The goal of the theory project is to provide an in-depth presentation of the paper's results, including its connections with the related literature. The report may be structured as follows

• Introduction and description of the problem
• Most important related works
• Notation and relevant definitions
• Proof of a technical result
• Some critical considerations.

Steps to complete the theory project:

1. Send an email directly to the instructor to agree on a topic;
2. Turn in the report via email to the instructor;
3. After 1-2 weeks, the instructor will get in touch to agree on a date for the oral discussion.

IMPORTANT FOR ALL PROJECTS: Your report must contain the following declaration: I/We declare that this material, which I/We now submit for assessment, is entirely my/our own work and has not been taken from the work of others, save and to the extent that such work has been cited and acknowledged within the text of my/our work. I/We understand that plagiarism, collusion, and copying are grave and serious offences in the university and accept the penalties that would be imposed should I engage in plagiarism, collusion or copying. This assignment, or any part of it, has not been previously submitted by me/us or any other person for assessment on this or any other course of study.

Course calendar:

Browse the calendar pages and click on a day to find out what was covered on that day.