9.520: Statistical Learning Theory and Applications, Spring 2008

Class Times:	Monday and Wednesday 10:30-12:00
Units:	3-0-9 H,G
Location:	46-5193
Instructors:	Tomaso Poggio (TP), Ryan Rifkin (RR), Jake Bouvrie (JB), Lorenzo Rosasco (LR)
Office Hours:	By appointment
Email Contact :	9.520@mit.edu
Previous Class:	SPRING 07

• Course Description
• Prerequisites
• Grading
• Problem Sets
• Projects
• Syllabus
• Reading List

Course description

The 2008 spring edition is an updated version of the course which has been running for several years. It focuses on the problem of supervised and unsupervised learning from the perspective of modern statistical learning theory, starting with the theory of multivariate function approximation from sparse data. Develops basic tools such as regularization, including support vector machines for regression and classification. Derives generalization bounds using stability. Discusses current research topics such as manifold regularization, feature selection, bayesian connections and techniques, and online learning. It emphasizes more than in previous years applications in several areas: computer vision, speech recognition and bioinformatics. It discusses advances in the neuroscience of cortex and their impact on learning theory and applications. Final projects and hands-on applications and exercises.

Prerequisites

6.867 or permission of instructor. In practice, a substantial level of mathematical maturity is necessary. Familiarity with probability and functional analysis will be very helpful. We try to keep the mathematical prerequisites to a minimum, but we will introduce complicated material at a fast pace.

Grading

There will be two problem sets, a Matlab assignment, and a final project. To receive credit, you must attend regularly, and put in effort on all problem sets and the project.

Problem sets

Problem set #1: PDF -- Due March XX
Problem set #2: PDF -- Due April XX

Projects

Project ideas: PDF

Syllabus

Follow the link for each class to find a detailed description, suggested readings, and class slides. Some of the later classes may be subject to reordering or rescheduling.

	Date	Title	Instructor(s)
Class 01	Wed 06 Feb	The Course at a Glance	TP
Class 02	Mon 11 Feb	The Learning Problem and Regularization Introduction to Fenchel Duality	TP RR
Class 03	Wed 13 Feb	Reproducing Kernel Hilbert Spaces Tikhonov Regularization, Value Regularization, and Fenchel Duality	LR RR
Mon 18 Feb - President's Day
Class 04	Tue 19 Feb	Regularized Least Squares	RR
Class 05	Wed 20 Feb	Several Views Of Support Vector Machines	RR
Class 06	Mon 25 Feb	Manifold Regularization	LR
Class 07	Wed 27 Feb	Sparse Approximation and Variable Selection	LR
Class 08	Mon 03 Mar	Iterative Optimization Techniques	Ross Lippert
Class 09	Wed 05 Mar	Generalization Bounds, Intro to Stability	Sasha Rakhlin
Class 10	Mon 10 Mar	Stability of Tikhonov Regularization	Sasha Rakhlin
Class 11	Wed 12 Mar	Bayesian Methods (TP's slides)	TP+Vikash
Class 12	Mon 17 Mar	Online Learning	Sanmay Das
Class 13	Wed 19 Mar	Loose ends, Project discussions
SPRING BREAK
Class 14	Mon 31 Mar	Multiclass Classification	RR
Class 15	Wed 02 Apr	Iterative Optimization Techniques	RR
Class 16	Mon 07 Apr	Vision and Visual Neuroscience	TP
Class 17	Wed 09 Apr	A Somewhat Unified Approach to Semi- and Un-supervised Learning	Ben Recht
Class 18	Mon 14 Apr	Learning in Circuits of Spiking Neurons - Hedonistic Synapses and Dynamic Conductance Perturbation	Sebastian Seung
Class 19	Wed 16 Apr	Computer Vision, Object Detection	Stan Bileschi
Mon 21 Apr - Patriot's Day
Class 20	Wed 23 Apr	Speech, Audio, and Auditory Neuroscience	JB
Class 21	Mon 28 Apr	Vision and Visual Neuroscience	Thomas Serre
Class 22	Wed 30 Apr	Energy-Based Models : the cure against Bayesian fundamentalism - slides | paper	Yann LeCun
Class 23	Mon 05 May	Conversion to the Bayesian Cult	Rev. Sayan Mukherjee
Class 24	Wed 07 May	Morphable Models for Video	Tony Ezzat
Class 25	Mon 12 May	Project Presentations
Class 26	Wed 14 May	Project Presentations
Math Camp 1	Functional analysis	XY
Math Camp 2	Probability theory	YX

Reading List

There is no textbook for this course. All the required information will be presented in the slides associated with each class. The books/papers listed below are useful general reference reading, especially from the theoretical viewpoint. A list of suggested readings will also be provided separately for each class.

Primary References

• S. Boucheron, O. Bousquet, and G. Lugosi, (2005), Theory of Classification: a Survey of Recent Advances. ESAIM: Probability and Statistics, 9:323-375.
• Bousquet, O., S. Boucheron and G. Lugosi. Introduction to Statistical Learning Theory. Advanced Lectures on Machine Learning Lecture Notes in Artificial Intelligence 3176, 169-207. (Eds.) Bousquet, O., U. von Luxburg and G. Ratsch, Springer, Heidelberg, Germany (2004)
• Bousquet, O.: New Approaches to Statistical Learning Theory. Annals of the Institute of Statistical Mathematics 55(2), 371-389 (2003)
• O. Bousquet and A. Elisseeff, Stability and Generalization, Journal of Machine Learning Research, Vol. 2, pp.499-526, 2002.
• F. Cucker and S. Smale. On The Mathematical Foundations of Learning. Bulletin of the American Mathematical Society, 2002.
• T. Evgeniou and M. Pontil and T. Poggio. Regularization Networks and Support Vector Machines. Advances in Computational Mathematics, 2000.
• R. Rifkin and R. Lippert, Value Regularization and Fenchel Duality, JMLR (to be published), 2007. (This webpage also contains many slides on Fenchel Duality, some of which will be used in the class.)
• T. Poggio and S. Smale. The Mathematics of Learning: Dealing with Data. Notices of the AMS, 2003

Secondary References

• L. Devroye, L. Gyorfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer, 1997.
• V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
• V. N. Vapnik. Statistical Learning Theory. Wiley, 1998.
• N. Cristianini and J. Shawe-Taylor. Introduction To Support Vector Machines. Cambridge, 2000.
• Poggio, T., R. Rifkin, S. Mukherjee and P. Niyogi. General Conditions for Predictivity in Learning Theory, Nature, Vol. 428, 419-422, 2004 (see also Past Performance and Future Results).

Background Mathematics References

• A. N. Kolmogorov and S. V. Fomin, Introductory Real Analysis, Dover Publications, 1975.
• A. N. Kolmogorov and S. V. Fomin, Elements of the Theory of Functions and Functional Analysis, Dover Publications, 1999.
• Luenberger, Optimization by Vector Space Methods, Wiley, 1969.