statistical methods for machine learning
6 CFU, Laurea magistrale in Informatica (F94)
machine learning
6 CFU, MSc in Data Science and Economics
INSTRUCTOR/DOCENTE: Nicolò Cesa-Bianchi |

statistical methods for machine learning
6 CFU, Laurea magistrale in Informatica (F94)
machine learning
6 CFU, MSc in Data Science and Economics
INSTRUCTOR/DOCENTE: Nicolò Cesa-Bianchi |

News

- Classes will start on March 1, 2021, with the following schedule: Monday 14:30-16:30 and Tuesday 9:30-11:30.
- For now, classes are via Zoom only. Go to the course page on Ariel to fetch the link. Videorecordings of all classes will be also made available through the same Ariel page.
- Student hours are offered via Zoom/Meet. Send an email to the instructor to make an appointment.

For students of the MSc in Data Science and Economics

- The course Machine learning, statistical learning, deep learning and artificial intelligence has
**two separate exams**, one for the MACHINE LEARNING module (Cesa-Bianchi, 40 hours, this course) and one for the STATISTICAL LEARNING module (Salini, 40 hours).

Bibliographic references:

Lecture notes provided by the instructor

The course makes heavy use of probability and statistics. A good textbook on these topics is:

Some good machine learning textbooks:Dimitri P. Bertsekas and John N. Tsitsiklis, Introduction to Probability (2nd edition). Athena Scientific, 2008.

Shai Shalev-Shwartz e Shai Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press, 2014.Mehryar Mohri, Afshin Rostamizadeh e Ameet Talwalkar, Foundations of Machine Learning, MIT Press, 2012.

L. Devroye, L. Gyorfi, and G. Lugosi, A Probabilistic Theory of Pattern Recognition, Springer, 1996.

Goals

Machine learning is concerned with the design of algorithms that can predict the evolution of a phenomenon based of a set of observations. A standard tool in the development of intelligent systems, machine learning has been successfully applied to a wide range of domains, including vision, speech and language, human-computer interaction, personalized recommendations, health and medicine, autonomous navigation, and many more. The course will describe and analyze, in a rigorous statistical framework, some of the most important machine learning techniques. This will provide the student with a rich set of methodological tools for understanding the general phenomenon of learning in machines.

**Note:** This is a course about the theoretical foundations of machine learning and the analysis of machine learning algorithms. The focus is on understanding the mathematical principles at the basis of machine learning.

Syllabus

- Introduction (version of April 26, 2020)
- The Nearest Neighbour algorithm (version of April 26, 2020)
- Tree predictors (version of April 26, 2020)
- Statistical learning (version of March 31, 2020)
- Hyperparameter tuning and risk estimates (version of April 2, 2020)
- Risk analysis for tree predictors (version of April 7, 2020)
- Risk analysis for Nearest Neighbour (version of October 3, 2020)
- Consistency and nonparametric algorithms (version of April 21, 2020)
- Linear prediction (version of April 28, 2020)
- Online learning (version of May 22, 2020)
- From sequential risk to statistical risk (version of May 11, 2020)
- Kernel functions (version of May 19, 2020)
- Support Vector Machines (version of June 10, 2020)
- Stability and risk control for SVM (version of June 4, 2020)
- Boosting and ensemble methods (version of June 8, 2020)
- Neural networks and deep learning (version of October 7, 2020: fixed typo in statement of Thm. 2)

Exams

- The current list of theory and experimental projects is valid until the end of May 2021.
- Students who choose the experimental project can do it jointly with the course of Algorithms for massive datasets (this option is offered only to students attending the MSc in Computer Science where both courses can be taken simultaneously).

The exam consists in writing a paper of about 10-15 pages containing either a report describing experimental results (experimental project) or a in-depth analysis of a theoretical topic (theory project). The steps each student must follow are:

- Write an email to the instructor to inform about which project among the ones offered you chose and wait for an approval
- Turn in the report by email to the instructor (if the project is experimental add a link to the code) and wait for the evaluation
- Once the report is evaluated, a date for the oral can be set
- At the oral, the student will be asked detailed questions on:
- the "core set" of the syllabus (lecture notes from 1 to 8 included)
- the lecture notes covering the algorithms used in the experiments, or the content of the report in case of a theory project

**IMPORTANT**

- Your report must contain the following declaration:
*I/We declare that this material, which I/We now submit for assessment, is entirely my/our own work and has not been taken from the work of others, save and to the extent that such work has been cited and acknowledged within the text of my/our work. I/We understand that plagiarism, collusion, and copying are grave and serious offences in the university and accept the penalties that would be imposed should I engage in plagiarism, collusion or copying. This assignment, or any part of it, has not been previously submitted by me/us or any other person for assessment on this or any other course of study.* - If your solution is adapted from other sources (e.g., Kaggle), this must be clearly stated, and the report should explain the differences and compare the experimental results.

The **experimental project** is typically based on implementing two or more learning algorithms (or variants of the same algorithm) from scratch. The algorithms are compared on real-world datasets. The programming language is immaterial. However, the implementation should be reasonable in terms of running time and memory footprint. If the experimental project is based on neural networks, then the student is allowed to use a toolbox (e.g., Tensorflow). The report, preferably written using LaTeX, will be evaluated according to the following criteria:

- Correctness of the general methodological approach
- Reproducibility of the experiments
- Correctness of the approach used for choosing the hyperparameters
- Clarity of exposition

The **theory project** is typically (but not exclusively) focused on a topic taught in class. The report will be based on one scientific paper (provided by the instructor), and must contain the complete proof of at least a technical result, including all necessary definitions and auxiliary lemmas. The goal of the theory project is to provide an in-depth presentation of the paper's results, including its connections with the related literature. The report may be structured as follows

- Introduction and description of the problem
- Most important related works
- Notation and relevant definitions
- Proof of a technical result
- Some critical considerations.

Projects:

- Current projects
- Joint project with the course
*Algorithms for massive datasets* - Older projects (valid through May 2020)

Browse the calendar pages and click on a day to find out what was covered on that day, and the link to the video in case the class was viderecorded.