9.520: Statistical Learning Theory and Applications, Spring 2003

Class Times: Monday and Wednesday 2:30-4:00
Location: E25-202
Instructors: Tomaso Poggio, Sayan Mukherjee, Ryan Rifkin, Alex Rakhlin
Office Hours: By appointment
Email Contact : 9.520@mit.edu

Course description

Focuses on the problem of supervised learning from the perspective of modern statistical learning theory starting with the problem of multivariate function approximation from sparse data. Develops basic tools such as Regularization including Support Vector Machines. Derives generalization bounds using both stability and metric entropy. Examines applications in several areas: computer vision, computer graphics, text classification and bioinformatics. A significant increase in final projects -- some of which may be seeds for theses -- and hands-on applications and exercises is planned, paralleling the rapidly increasing practical uses of the techniques described in the subject.

Prerequisites

18.02, 9.641, 6.893 or permission of instructor. In practice, a substantial level of mathematical maturity is necessary. Familiarity with probability and functional analysis will be very helpful. We try to keep the mathematical prerequisites to a minimum, but we will introduce complicated material at a fast pace.

Grading

There will be two problem sets, a Matlab assignment, and a final project. To receive credit, you must attend regularly, and put in effort on all problem sets and the project.

Problem sets

Problem set #1: PS, PDF
Problem set #2: PS, PDF

Projects

Project ideas: PS, PDF

Syllabus

Follow the link for each class to find a detailed description, suggested readings, and class slides. Some of the later classes may be subject to reordering or rescheduling.



Date Title Instructor(s)
Class 01 Wed 05 Feb The Course at a Glance TP
Class 02 Mon 10 Feb The Learning Problem in Perspective TP
Class 03 Wed 12 Feb Regularization and Reproducing Kernel Hilbert Spaces TP,SM
No Class Tue 18 Feb Snow, lots of it Yeti
Class 04 Wed 19 Feb Regression and Least-Squares Classification RR
Class 05 Mon 24 Feb Support Vector Machines for Classification RR
Class 06 Wed 26 Feb Generalization Bounds, Intro to Stability AR
Class 07 Mon 03 Mar Stability of Tikhonov Regularization (slides updated!) AR
Class 08 Wed 05 Mar Consistency and Uniform Convergence Over Function Classes SM,AR
Class 09 Mon 10 Mar Necessary and sufficient conditions for Uniform Convergence SM,AR
Class 10 Wed 12 Mar Bagging and Boosting SM
Class 11 Mon 17 Mar Computer Vision, Object Detection BH,SB
Class 12 Wed 19 Mar Loose Ends TP,SM,RR,AR
SPRING BREAK
Class 13 Mon 31 Mar Approximation Theory FG
Class 14 Wed 02 Apr RKHS, Mercer Thm, Unbounded Domains, Frames and Wavelets TP,SM
Class 15 Mon 07 Apr Bioinformatics GY,SM
Class 16 Wed 09 Apr Text JR
Class 17 Mon 14 Apr Regularization Networks TP
Class 18 Wed 16 Apr Morphable Models for Video TE
Class 19 Wed 23 Apr Leave-one-out approximations SM
Class 20 Mon 28 Apr Bayesian Interpretations TP,SM
Class 21 Wed 30 Apr Multiclass Classification RR
Class 22 Mon 05 May Stability and Glivenko-Cantelli Classes TP,SM
Class 23 Wed 07 May Symmetrization, Rademacher Averages AR,SM
Class 24 Mon 12 May Project Presentations
Class 25 Wed 14 May Project Presentations

Math Camp TBA Lagrange Multipliers/Convex Optimization RR
Math Camp Wed 12,19 Feb Functional Analysis SM
Extra Topic TBA SVM Rules of Thumb RR

Reading List

There is no textbook for this course. All the required information will be presented in the slides associated with each class. The books listed below are useful general reference reading, especially from the theoretical viewpoint. A list of suggested readings will also be provided separately for each class.