9.520: Statistical Learning Theory and Applications, Spring 2006

Class Times: Monday and Wednesday 10:30-12:00
Units: 3-0-9 H,G
Location: 46-5056
Instructors: Tomaso Poggio (TP), Sasha Rakhlin (AR), Andrea Caponnetto (AC), Ryan Rifkin (RR)
Office Hours: By appointment
Email Contact : 9.520@mit.edu
Previous Classes: SPRING 04

Course description

Focuses on the problem of supervised and unsupervised learning from the perspective of modern statistical learning theory, starting with the theory of multivariate function approximation from sparse data. Develops basic tools such as regularization, including support vector machines for regression and classification. Derives generalization bounds using both stability and VC theory. Discusses current research topics such as boosting, feature selection, active learning, ranking, and online learning. Examines applications in several areas: computer vision, computer graphics and bioinformatics. Final projects and hands-on applications and exercises, paralleling the rapidly increasing practical uses of the techniques described in the subject.

Prerequisites

18.02, 9.641, 6.893 or permission of instructor. In practice, a substantial level of mathematical maturity is necessary. Familiarity with probability and functional analysis will be very helpful. We try to keep the mathematical prerequisites to a minimum, but we will introduce complicated material at a fast pace.

Grading

There will be two problem sets, a Matlab assignment, and a final project. To receive credit, you must attend regularly, and put in effort on all problem sets and the project.

Problem sets

Problem set #1: PS, PDF. -- Due March 20.
Problem set #2: PS, PDF. -- Due April 26.

Projects

Project ideas: PS, PDF

Syllabus

Follow the link for each class to find a detailed description, suggested readings, and class slides. Some of the later classes may be subject to reordering or rescheduling.



Date Title Instructor(s)
Class 01 Wed 08 Feb The Course at a Glance TP
Class 02 Mon 13 Feb The Learning Problem in Perspective TP
Class 03 Wed 15 Feb Reproducing Kernel Hilbert Spaces AC
Class 04 Tue 21 Feb Regression and Least-Squares Classification RR
Class 05 Wed 22 Feb Support Vector Machines for Classification RR
Class 6 Mon 27 Feb Manifold regularization AC
Class 7 Wed 01 Mar Unsupervised Learning Techniques AC
Class 8 Mon 06 Mar Multiclass RR
Class 9 Wed 08 Mar Ranking Giorgos Zacharia
Class 10 Mon 13 Mar Boosting and Bagging AR
Class 11 Wed 15 Mar Computer Vision, Object Detection Stan Bileschi
Class 12 Mon 20 Mar Online Learning Sanmay Das and AC
Class 13 Wed 22 Mar Loose ends, Project discussions
SPRING BREAK
Class 14 Mon 03 Apr Generalization Bounds, Intro to Stability AR
Class 15 Wed 05 Apr Stability of Tikhonov Regularization AR
Class 16 Mon 10 Apr Uniform Convergence Over Function Classes AR
Class 17 Wed 12 Apr Uniform Convergence for Classification. VC-dimension. AR
Class 18 Wed 19 Apr Neuroscience Thomas Serre
Class 19 Mon 24 Apr Symmetrization, Rademacher Averages AR
Class 20 Wed 26 Apr Fenchel Duality Ross Lippert and RR
Class 21 Mon 01 May Speech/Audio Jake Bouvrie
Class 22 Wed 03 May Active learning Claire Monteleoni
Class 23 Mon 08 May Morphable Models for Video Tony Ezzat
Class 24 Wed 10 May Bioinformatics Sayan Mukherjee
Class 25 Mon 15 May Project Presentations
Class 26 Wed 17 May Project Presentations

Math Camp 1 Mon 13 Feb Functional analysis AC
Math Camp 2 Tue 21 Feb Probability theory AR

Reading List

There is no textbook for this course. All the required information will be presented in the slides associated with each class. The books/papers listed below are useful general reference reading, especially from the theoretical viewpoint. A list of suggested readings will also be provided separately for each class.