Course Description
This course will focus on theoretical aspects of Statistical Learning and Sequential Prediction (Online Learning). In the first part of the course, we will analyze learning with i.i.d. data using classical tools: concentration inequalities, random averages, covering numbers, and combinatorial parameters. We then focus on sequential prediction and develop many of the same tools for learning in this scenario. The latter part is based on recent research and offers many directions for further investigation. The minimax approach, which we emphasize throughout the course, offers a systematic way of comparing learning problems. Beyond the theoretical analysis, we will discuss learning algorithms and, in particular, an important connection between learning and optimization. Our framework will give a handle on developing near-optimal and computationally efficient algorithms. We will illustrate this on the problems of matrix completion, link prediction, and other. Time permitting, we will make excursions into Information Theory and Game Theory, and show how our new tools seamlessly yield a number of interesting results.
Prerequisites: Probability Theory and Linear Algebra.
Lecture Notes
These
lecture notes are constantly evolving, so if your version says x<y today, it might say x>y tomorrow.
The "Algorithms" section will go through an overhaul this semester.