9.520: Statistical Learning Theory and Applications, Fall 2013

Class Times:	Monday and Wednesday
Units:	3-0-9 H,G
Location:	46-5193
Instructors:	Tomaso Poggio (TP), Lorenzo Rosasco (LR), Carlo Ciliberto (CC).
Office Hours:	Friday 1-2 pm in 46-5156, CBCL lounge (by appointment)
Email Contact :	9.520@mit.edu
Previous Class:	SPRING 12

• Course Description
• Prerequisites
• Grading
• Scribe Notes
• Problem Sets
• Projects
• Syllabus
• Reading List

Course description

The class introduces the theory and algorithms of computational learning in the framework of statistics and functional analysis. It gives an in-depth discussion of state of the art machine learning algorithms, for regression and classification, variable selection, manifold learning and transfer learning. The class focuses on the unifying role of regularization.

Many problems in applied science are inverse problems and most inverse problems are ill-posed: the solution does not satisfy the basic requirement of existence, uniqueness and stability. As it turns out most sensory problems are inverse and ill-posed problems. In a sense, intelligence is the ability of solving effectively inverse problems. Probably the most interesting inverse and ill-posed problem -- and the one which is at the very core of intelligence -- is the problem of learning from experience.

The theory and algorithms of regularization provide principled ways to solve ill-posed problems and restore well-posedness. Not surprisingly, most of the successful machine learning algorithms, such as MobilEye's vision system for cars and new system for intelligent assistants, are based on regularization techniques.

The goal of this class is to provide students with the knowledge needed to use and develop effective computational learning solutions to challenging problems.

Prerequisites

We will make extensive use of linear algebra, basic functional analysis (we cover the essentials in class and during the math-camp), basic concepts in probability theory and concentration of measure (also covered in class and during the mathcamp). Students are expected to be familiar with Matlab.

Grading

Requirements for grading (other than attending lectures) are: scribing one lecture, 2 problems sets, and a final project.

Scribe Notes

In this class, there will be three to five unscribed lectures; of the remaining lectures, new scribe notes for classes #5,6,9,11 will be created, while those of lectures #2 - #8, #10, #12, and lectures #14 - #18 will be edited from existing notes. Each student taking the class for credit will be required to work on improving and updating, or creating the scribe notes of one lecture. Scribe notes should be a natural integration of the presentation of the lectures with the material in the slides. The lecture slides are available on this website for your reference. Good scribe notes are important both for your grades, and for other students to read. In particular, please make an effort to present the material in a clear, concise, and comprehensive manner.

Scribe notes must be prepared with Latex, using the provided template. Scribe notes (.tex file and all additional files) should be submitted to 9.520@mit.edu no later than one week after the class. Please make sure to proofread the notes carefully before submitting. We will review the scribe notes to check the technical content and quality of writing. We will also give feedback and ask for a revised version if necessary. Completed scribe notes will be posted on this website as soon as possible.

Problem Sets

Problem set #1
Problem set #2

Projects

The final project can be either a wikipedia entry or a research project (we recommend a Wikipedia entry).
We envision 2 kinds of research project:

• Applications: evaluate an algorithm on some interesting problem of your choice;
• Theory and Algorithms: study theoretically or empirically some new machine learning algorithm/problem.

For the Wikipedia article, we suggest a short one using the Wikipedia standard article format; for the research project you should use this template. Reports should be 8 pages maximum, including references. Additional material can be included in the appendix.

Previous project suggestions: Spring 2012 projects

Syllabus

Follow the link for each class to find a detailed description, suggested readings, and class slides. Some of the later classes may be subject to reordering or rescheduling.

Class	Date	Title	Instructor(s)
Class 01	Wed 04 Sep	The Course at a Glance	TP
Class 02	Mon 09 Sep	The Learning Problem and Regularization	TP
Class 03	Wed 11 Sep	Reproducing Kernel Hilbert Spaces	LR
Class 04	Mon 16 Sep	Mercer Theorem and Feature Maps	LR
Class 05	Wed 18 Sep	Tikhonov Regularization and the Representer Theorem	LR
Class 06	Mon 23 Sep	Regularized Least Squares and Support Vector Machines	LR
Class 07	Wed 25 Sep	Generalization Bounds, Intro to Stability	LR/TP
Class 08	Mon 30 Sep	Stability of Tikhonov Regularization	LR/TP
Class 09	Wed 2 Oct	Regularization Parameter Choice: Theory and Practice	LR
Class 10	Mon 07 Oct	Bayesian Interpretations of Regularization	LR
Class 11	Wed 09 Oct	Spectral Regularization	LR
Monday 14 October - Columbus Day
Class 12	Wed 16 Oct	Regularization for Multi-Output Learning	LR
Class 13	Thurs 17 Oct	Loose ends, Project discussions
Class 14	Mon 21 Oct	Sparsity Based Regularization 1	LR
Class 15	Wed 23 Oct	Sparsity Based Regularization 2	LR
Class 16	Mon 28 Oct	Regularization with Multiple Kernels	LR
Class 17	Wed 30 Oct	On-line Learning	LR
Class 18	Mon 04 Nov	Manifold Regularization	LR
Class 19	Wed 06 Nov	Hierarchical Representation for Learning: Visual Cortex	TP
Monday 11 November - Veterans Day
Class 20	Tues 12 Nov	Hierarchical Representation for Learning: Mathematics	LR
Class 21	Wed 13 Nov	Hierarchical Representation for Learning: Computational Model	TP
Class 22	Mon 18 Nov	Learning Data Representation with Regularization	LR
Class 23	Wed 20 Nov	TBA
Class 24	Mon 25 Nov	TBA
Class 25	Wed 27 Nov	Project Presentations
Class 26	Mon 02 Dec	Project Presentations

Reading List

There is no textbook for this course. All the required information will be presented in the slides associated with each class. The books/papers listed below are useful general reference reading, especially from the theoretical viewpoint. A list of suggested readings will also be provided separately for each class.

Primary References

• Bousquet, O., S. Boucheron and G. Lugosi. Introduction to Statistical Learning Theory. Advanced Lectures on Machine Learning Lecture Notes in Artificial Intelligence 3176, 169-207. (Eds.) Bousquet, O., U. von Luxburg and G. Ratsch, Springer, Heidelberg, Germany (2004)
• F. Cucker and S. Smale. On The Mathematical Foundations of Learning. Bulletin of the American Mathematical Society, 2002.
• F. Cucker and D-X. Zhou. Learning theory: an approximation theory viewpoint. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, Cambridge, 2007.
• L. Devroye, L. Gyorfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer, 1997.
• T. Evgeniou and M. Pontil and T. Poggio. Regularization Networks and Support Vector Machines. Advances in Computational Mathematics, 2000.
• T. Poggio and S. Smale. The Mathematics of Learning: Dealing with Data. Notices of the AMS, 2003
• I. Steinwart and A. Christmann. Support vector machines. Springer, New York, 2008.
• V. N. Vapnik. Statistical Learning Theory. Wiley, 1998.
• V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
• N. Cristianini and J. Shawe-Taylor. Introduction To Support Vector Machines. Cambridge, 2000.

Background Mathematics References

• A. N. Kolmogorov and S. V. Fomin, Introductory Real Analysis, Dover Publications, 1975.
• A. N. Kolmogorov and S. V. Fomin, Elements of the Theory of Functions and Functional Analysis, Dover Publications, 1999.
• Luenberger, Optimization by Vector Space Methods, Wiley, 1969.

Neuroscience Related References

• Serre, T., L. Wolf, S. Bileschi, M. Riesenhuber and T. Poggio. "Object Recognition with Cortex-like Mechanisms", IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 3, 411-426, 2007.
• Serre, T., A. Oliva and T. Poggio."A Feedforward Architecture Accounts for Rapid Categorization", Proceedings of the National Academy of Sciences (PNAS), Vol. 104, No. 15, 6424-6429, 2007.
• S. Smale, L. Rosasco, J. Bouvrie, A. Caponnetto, and T. Poggio. "Mathematics of the Neural Response", Foundations of Computational Mathematics, Vol. 10, 1, 67-91, June 2009.