9.520: Statistical Learning Theory and Applications, Spring 2004

Class Times: Monday and Wednesday 10:30-12:00
Units: 3-0-9 H,G
Location: E25-202
Instructors: Tomaso Poggio, Sayan Mukherjee, Ryan Rifkin, Alex Rakhlin
Office Hours: By appointment
Email Contact : 9.520@mit.edu
Previous Classes: SPRING 03

Course description

Focuses on the problem of supervised learning from the perspective of modern statistical learning theory starting with the problem of multivariate function approximation from sparse data. Develops basic tools such as Regularization including Support Vector Machines. Derives generalization bounds using both stability and covering number conditions. Describes applications in several areas, such as computer vision, computer graphics, text classification and bioinformatics. Several final projects -- some of which may be seeds for theses -- and hands-on applications and exercises are planned, given the rapidly increasing practical use of the techniques described in the subject.

Prerequisites

18.02, 9.641, 6.893 or permission of instructor. In practice, a substantial level of mathematical maturity is necessary. Familiarity with probability and functional analysis will be very helpful. We try to keep the mathematical prerequisites to a minimum, but we will introduce complicated material at a fast pace.

Grading

There will be two problem sets, a Matlab assignment, and a final project. To receive credit, you must attend regularly, and put in effort on all problem sets and the project.

Problem sets

Problem set #1: PS, PDF (updated 3/8/2004)
Problem set #2: PS, PDF

Projects

Project ideas: PS, PDF

Syllabus

Follow the link for each class to find a detailed description, suggested readings, and class slides. Some of the later classes may be subject to reordering or rescheduling.



Date Title Instructor(s)
Class 01 Wed 04 Feb The Course at a Glance TP
Class 02 Mon 9 Feb The Learning Problem in Perspective TP
Class 03 Wed 11 Feb Regularization and Reproducing Kernel Hilbert Spaces TP,SM
Class 04 Tue 17 Feb Regression and Least-Squares Classification RR
Class 05 Wed 18 Feb Support Vector Machines for Classification RR
Class 06 Mon 23 Feb Generalization Bounds, Intro to Stability AR
Class 07 Wed 26 Feb Stability of Tikhonov Regularization AR
Class 08 Mon 01 Mar Consistency and Uniform Convergence Over Function Classes AR
Class 09 Wed 03 Mar Necessary and sufficient conditions for Uniform Convergence SM
Class 10 Mon 8 Mar Stability and Glivenko-Cantelli Classes TP
Class 11 Wed 10 Mar Multiclass Classification RR
Class 12 Mon 15 Mar Computer Vision, Object Detection LW,SB
Class 13 Wed 17 Mar Loose ends, Project discussions TP,SM,RR,AR
SPRING BREAK
Class 14 Mon 29 Mar Boosting and Bagging SM
Class 15 Wed 31 Mar Text JR
Class 16 Mon 05 Apr Approximation Theory FG
Class 17 Wed 07 Apr Symmetrization, Rademacher Averages AR
Class 18 Mon 12 Apr Regularization Networks TP
Class 19 Wed 14 Apr Morphable Models for Video TE
Class 20 Wed 21 Apr Leave-one-out approximations SM
Class 21 Mon 26 Apr Computational biology GY,SM
Class 22 Wed 28 Apr RKHS, Mercer Thm, Unbounded Domains, Frames and Wavelets TP,SM
Class 23 Mon 03 May Unsupervised Learning, Learning with Partially Labeled Data,
Active Learning, Learning Manifolds		 MB
Class 24 Wed 05 May Bayesian Interpretations TP,SM
Class 25 Mon 10 May Project Presentations
Class 26 Wed 12 May Project Presentations

Math Camp 1 Mon 9 Feb Analysis and basic probability theory SM
Math Camp 2 Tue 17 Feb More analysis and probability theory (Updated!) TP,AR

Reading List

There is no textbook for this course. All the required information will be presented in the slides associated with each class. The books listed below are useful general reference reading, especially from the theoretical viewpoint. A list of suggested readings will also be provided separately for each class.