Class Times: Monday and Wednesday 10:30-12:00 Units: 3-0-9 H,G Location: 46-5193 Instructors: Tomaso Poggio (TP), Ryan Rifkin (RR), Jake Bouvrie (JB), Lorenzo Rosasco (LR)
Office Hours: By appointment Email Contact : 9.520@mit.edu Previous Class: SPRING 06 Course description
The 2007 spring edition is an updated version of the course which has been running for several years. It focuses on the problem of supervised and unsupervised learning from the perspective of modern statistical learning theory, starting with the theory of multivariate function approximation from sparse data. Develops basic tools such as regularization, including support vector machines for regression and classification. Derives generalization bounds using stability. Discusses current research topics such as manifold regularization, feature selection, bayesian connections and techniques, and online learning. It emphasizes more than in previous years applications in several areas: computer vision, speech recognition and bioinformatics. It discusses advances in the neuroscience of cortex and their impact on learning theory and applications. Final projects and hands-on applications and exercises.Prerequisites
6.867 or permission of instructor. In practice, a substantial level of mathematical maturity is necessary. Familiarity with probability and functional analysis will be very helpful. We try to keep the mathematical prerequisites to a minimum, but we will introduce complicated material at a fast pace.Grading
There will be two problem sets, a Matlab assignment, and a final project. To receive credit, you must attend regularly, and put in effort on all problem sets and the project.
Problem sets
Problem set #1: PDF -- Due March 19th.
Problem set #2: PDF -- Due Mon., April 23rd.
Projects
Project ideas: PDF (list as of 03/01/07)
Syllabus
Follow the link for each class to find a detailed description, suggested readings, and class slides. Some of the later classes may be subject to reordering or rescheduling.
Date Title Instructor(s) Class 01 Wed 06 Feb The Course at a Glance TP Class 02 Mon 12 Feb The Learning Problem and Regularization
Introduction to Fenchel DualityTP
RRClass 03 Wed 14 Feb Reproducing Kernel Hilbert Spaces
Tikhonov Regularization, Value Regularization, and Fenchel DualityLR
RRMon 18 Feb - President's Day Class 04 Tue 20 Feb Regularized Least Squares RR Class 05 Wed 21 Feb Several Views Of Support Vector Machines RR Class 06 Mon 26 Feb Manifold Regularization LR Class 07 Wed 28 Feb Sparse Approximation and Variable Selection LR Class 08 Mon 05 Mar Iterative Optimization Techniques Ross Lippert Class 09 Wed 07 Mar Generalization Bounds, Intro to Stability Sasha Rakhlin Class 10 Mon 12 Mar Stability of Tikhonov Regularization Sasha Rakhlin Class 11 Wed 14 Mar Bayesian Methods (TP's slides) TP+Vikash Class 12 Mon 19 Mar Online Learning Sanmay Das Class 13 Wed 21 Mar Loose ends, Project discussions SPRING BREAK Class 14 Mon 02 Apr Multiclass Classification RR Class 15 Wed 04 Apr Iterative Optimization Techniques RR Class 16 Mon 09 Apr Vision and Visual Neuroscience TP Class 17 Wed 11 Apr A Somewhat Unified Approach to Semi- and Un-supervised Learning Ben Recht Class 18 Wed 18 Apr Learning in Circuits of Spiking Neurons - Hedonistic Synapses and Dynamic Conductance Perturbation Sebastian Seung Class 19 Mon 23 Apr Computer Vision, Object Detection Stan Bileschi Class 20 Wed 25 Apr Speech, Audio, and Auditory Neuroscience JB Class 21 Mon 30 Apr Vision and Visual Neuroscience Thomas Serre Class 22 Wed 02 May Energy-Based Models : the cure against Bayesian fundamentalism - slides | paper Yann LeCun Class 23 Mon 07 May Conversion to the Bayesian Cult Rev. Sayan Mukherjee Class 24 Wed 09 May Morphable Models for Video Tony Ezzat Class 25 Mon 14 May Project Presentations Class 26 Wed 16 May Project Presentations
Math Camp 1 Functional analysis XY Math Camp 2 Probability theory YX Reading List
There is no textbook for this course. All the required information will be presented in the slides associated with each class. The books/papers listed below are useful general reference reading, especially from the theoretical viewpoint. A list of suggested readings will also be provided separately for each class.Primary References
- S. Boucheron, O. Bousquet, and G. Lugosi, (2005), Theory of Classification: a Survey of Recent Advances. ESAIM: Probability and Statistics, 9:323-375.
- Bousquet, O., S. Boucheron and G. Lugosi. Introduction to Statistical Learning Theory. Advanced Lectures on Machine Learning Lecture Notes in Artificial Intelligence 3176, 169-207. (Eds.) Bousquet, O., U. von Luxburg and G. Ratsch, Springer, Heidelberg, Germany (2004)
- Bousquet, O.: New Approaches to Statistical Learning Theory. Annals of the Institute of Statistical Mathematics 55(2), 371-389 (2003)
- O. Bousquet and A. Elisseeff, Stability and Generalization, Journal of Machine Learning Research, Vol. 2, pp.499-526, 2002.
- F. Cucker and S. Smale. On The Mathematical Foundations of Learning. Bulletin of the American Mathematical Society, 2002.
- T. Evgeniou and M. Pontil and T. Poggio. Regularization Networks and Support Vector Machines. Advances in Computational Mathematics, 2000.
- R. Rifkin and R. Lippert, Value Regularization and Fenchel Duality, JMLR (to be published), 2007. (This webpage also contains many slides on Fenchel Duality, some of which will be used in the class.)
- T. Poggio and S. Smale. The Mathematics of Learning: Dealing with Data. Notices of the AMS, 2003
Secondary References
- L. Devroye, L. Gyorfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer, 1997.
- V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
- V. N. Vapnik. Statistical Learning Theory. Wiley, 1998.
- N. Cristianini and J. Shawe-Taylor. Introduction To Support Vector Machines. Cambridge, 2000.
- Poggio, T., R. Rifkin, S. Mukherjee and P. Niyogi. General Conditions for Predictivity in Learning Theory, Nature, Vol. 428, 419-422, 2004 (see also Past Performance and Future Results).
Background Mathematics References
- A. N. Kolmogorov and S. V. Fomin, Introductory Real Analysis, Dover Publications, 1975.
- A. N. Kolmogorov and S. V. Fomin, Elements of the Theory of Functions and Functional Analysis, Dover Publications, 1999.
- Luenberger, Optimization by Vector Space Methods, Wiley, 1969.