reinforcement learning laboratory

3 CFU, MSc in Data Science for Economics

Instructors: Nicolò Cesa-Bianchi, Alfio Ferrara


The laboratory will start on January 12, 2023.


This laboratory introduces the theoretical and algorithmic foundations of Reinforcement Learning, the subfield of Machine Learning studying adaptive agents that take actions and interact with an unknown environment. Reinforcement learning is a powerful paradigm for the study of autonomous AI systems, and has been applied to a wide range of tasks including autonomous driving, industrial automation, conversational agents, trading and finance, game playing, and healthcare.


  1. Introduction (version Jan 20, 2023) — 2 classes
    1. What is reinforcement learning
    2. Markov decision processes
    3. Evaluation criteria: finite horizon, infinite horizon, discounted horizon
    4. Markov policies and their properties
  2. Finite horizon (version Jan 19, 2023) — 1 class
    1. State-value function
    2. Action-value function
    3. Bellman optimality equations for finite horizon
  3. Discounted horizon (version Jan 20, 2023) — 1 class
    1. Bellman optimality equations for discounted horizon
    2. Value iteration
    3. Policy iteration
    4. Linear programming interpretation
  4. Model-Free reinforcement learning — 2 classes
    1. Q-learning
    2. SARSA
  5. Temporal difference algorithms — 2 classes
    1. TD(0)
    2. TD(λ)
    3. Equivalence between forward and backward view
    4. SARSA(λ)
  6. Developing a reinforcement learning project — 2 classes
Reference material


The exam consists in developing an experimental project and writing a report which will be discussed in the oral exam. The discussion will also include questions on the theory covered in the course. The final grade will take into account both the project and the oral exam.

Course calendar:

Browse the calendar pages to find out what was covered in each class.