9.520: Statistical Learning Theory and Applications, Spring 2010


Class Times: Monday and Wednesday 10:30-12:00
Units: 3-0-9 H,G
Location: 46-5193
Instructors: Tomaso Poggio (TP), Lorenzo Rosasco (LR), Charlie Frogner (CF)
TA: Andre Wibisono
Office Hours: Friday 1-2 pm in 46-5165
Email Contact : 9.520@mit.edu
Previous Class: SPRING 09

Sign up for final project presentation here.

Course description

Focuses on the problem of supervised and unsupervised learning from the perspective of modern statistical learning theory, starting with the theory of multivariate function approximation from sparse data. Develops basic tools such as regularization, including support vector machines for regression and classification. Derives generalization bounds using stability. Discusses current research topics such as manifold regularization, sparsity, feature selection, Bayesian connections and techniques, and online learning. Emphasizes applications in several areas: computer vision, speech recognition, and bioinformatics. Discusses advances in the neuroscience of the cortex and their impact on learning theory and applications. Requirements for grading (other than attending lectures) are: scribing one lecture, 2 problems sets, final project.

Prerequisites

6.867 or permission of instructor. In practice, a substantial level of mathematical maturity is necessary. Familiarity with probability and functional analysis will be very helpful. We try to keep the mathematical prerequisites to a minimum, but we will introduce complicated material at a fast pace.

Grading

Requirements for grading (other than attending lectures) are: scribing one lecture, 2 problems sets, final project.

Scribe Notes

In this class we will scribe 13 lectures: lectures #2 - #11, and lectures #14 - #16. Each student who is taking this class for credit will be required to scribe one lecture. Scribe notes should be a natural integration of the presentation of the lectures with the material in the slides. The lecture slides are available on this website for your reference. Good scribe notes are important, both for your grades and for other students to read. In particular, please make an effort to present the material in a clear, concise, and comprehensive manner. You can look at the lecture notes of 6.854/18.415 (Fall 2008) on OCW to see examples of the expected format and quality of scribe notes.

Scribe notes must be prepared with Latex, using this template. Scribe notes (.tex file and all additional files) should be submitted to 9.520@mit.edu no later than one week after the class. Please make sure to proofread the notes carefully before submitting. We will review the scribe notes to check the technical content and quality of writing. We will also give feedback and ask for a revised version if necessary. Completed scribe notes will be posted on this website as soon as possible.

You can sign up here for scribe notes. If you have problems opening or editing the page, please send us an email at 9.520@mit.edu. In addition, if you have any questions of concerns about the scribing requirement, please feel free to send us an email.

Problem Sets

Problem set #1: PDF | two moons dataset -- due Monday, March 15th
Problem set #2: PDF -- due Wednesday, April 14th UPDATED 4/6

Projects

Project ideas: PDF
Abstract Proposals due Monday, March 29th

Syllabus

Follow the link for each class to find a detailed description, suggested readings, and class slides. Some of the later classes may be subject to reordering or rescheduling.

 


Date Title Instructor(s) Scribe notes
Class 01 Wed 03 Feb The Course at a Glance TP ---
Class 02 Mon 08 Feb The Learning Problem and Regularization TP ---
Class 03 Wed 10 Feb Reproducing Kernel Hilbert Spaces LR PDF
Mon 15 Feb - President's Day
Class 04 Tue 16 Feb Regularized Least Squares CF PDF
Class 05 Wed 17 Feb Several Views Of Support Vector Machines CF PDF
Class 06 Mon 22 Feb Generalization Bounds, Intro to Stability LR/TP PDF
Class 07 Wed 24 Feb Stability of Tikhonov Regularization LR/TP ---
Class 08 Mon 01 Mar Spectral Regularization LR ---
Class 09 Wed 03 Mar Manifold Regularization LR PDF
Class 10 Mon 08 Mar Sparsity Based Regularization LR PDF
Class 11 Wed 10 Mar Regularization Methods for Multi-Output Learning LR PDF
Class 12 Mon 15 Mar Regularization Methods for Online Learning Sasha Rakhlin ---
Class 13 Wed 17 Mar Loose ends, Project discussions TP/LR ---
SPRING BREAK March 19-25
Class 14 Mon 29 Mar Vision and Visual Neuroscience TP ---
Class 15 Wed 31 Mar Vision and Visual Neuroscience (HMAX) TP ---
Class 16 Mon 05 Apr Derived Kernels LR ---
Class 17 Wed 07 Apr Application of Belief Nets to Modelling Attention Sharat Chikkerur ---
Class 18 Mon 12 Apr Active Learning: Closing the loop between data analysis and aquisition Rui Castro ---
Class 19 Wed 14 Apr Deep Boltzman Machines Ruslan Salakhtudinov ---
Mon 19 Apr - Patriot's Day
Class 20 Wed 21 Apr Bayesian Interpretations of Regularization CF ---
Class 21 Mon 26 Apr Bayesian Learning and Nonparametric Bayesian Methods LR ---
Class 22 Wed 28 Apr Approximate Inference Ruslan Salakhtudinov ---
Class 23 Mon 03 May Multiple Kernel Learning LR ---
Class 24 Wed 05 May Geometry and Analysis of Data Sets in High Dimension Mauro Maggioni ---
Class 25 Mon 10 May Project Presentations --- ---
Class 26 Wed 12 May Project Presentations --- ---

Math Camp Mon 08 Feb Functional analysis: slides, notes. Probability theory: notes --- ---

Reading List

There is no textbook for this course. All the required information will be presented in the slides associated with each class. The books/papers listed below are useful general reference reading, especially from the theoretical viewpoint. A list of suggested readings will also be provided separately for each class.

Primary References

Secondary References

Background Mathematics References

Neuroscience Related References