Class Times: Monday and Wednesday Units: 3-0-9 H,G Location: 46-5193 Instructors: Office Hours: Friday 1-2 pm in 46-5156, CBCL lounge (by appointment) Email Contact : firstname.lastname@example.org Previous Class: SPRING 12
The class introduces the theory and algorithms of computational learning in the framework of statistics and functional analysis. It gives an in-depth discussion of state of the art machine learning algorithms, for regression and classification, variable selection, manifold learning and transfer learning. The class focuses on the unifying role of regularization.
Many problems in applied science are inverse problems and most inverse problems are ill-posed: the solution does not satisfy the basic requirement of existence, uniqueness and stability. As it turns out most sensory problems are inverse and ill-posed problems. In a sense, intelligence is the ability of solving effectively inverse problems. Probably the most interesting inverse and ill-posed problem -- and the one which is at the very core of intelligence -- is the problem of learning from experience.
The theory and algorithms of regularization provide principled ways to solve ill-posed problems and restore well-posedness. Not surprisingly, most of the successful machine learning algorithms, such as MobilEye's vision system for cars and new system for intelligent assistants, are based on regularization techniques.
The goal of this class is to provide students with the knowledge needed to use and develop effective computational learning solutions to challenging problems.
PrerequisitesWe will make extensive use of linear algebra, basic functional analysis (we cover the essentials in class and during the math-camp), basic concepts in probability theory and concentration of measure (also covered in class and during the mathcamp). Students are expected to be familiar with Matlab.
GradingRequirements for grading (other than attending lectures) are: scribing one lecture, 2 problems sets, and a final project.
In this class, there will be three to five unscribed lectures; of the remaining lectures, new scribe notes for classes #5,6,9,11 will be created, while those of lectures #2 - #8, #10, #12, and lectures #14 - #18 will be edited from existing notes. Each student taking the class for credit will be required to work on improving and updating, or creating the scribe notes of one lecture. Scribe notes should be a natural integration of the presentation of the lectures with the material in the slides. The lecture slides are available on this website for your reference. Good scribe notes are important both for your grades, and for other students to read. In particular, please make an effort to present the material in a clear, concise, and comprehensive manner.
Scribe notes must be prepared with Latex, using the provided template. Scribe notes (.tex file and all additional files) should be submitted to email@example.com no later than one week after the class. Please make sure to proofread the notes carefully before submitting. We will review the scribe notes to check the technical content and quality of writing. We will also give feedback and ask for a revised version if necessary. Completed scribe notes will be posted on this website as soon as possible.
Problem SetsProblem set #1 Problem set #2
ProjectsThe final project can be either a wikipedia entry or a research project (we recommend a Wikipedia entry).
We envision 2 kinds of research project:
For the Wikipedia article, we suggest a short one using the Wikipedia standard article format; for the research project you should use this template. Reports should be 8 pages maximum, including references. Additional material can be included in the appendix.
- Applications: evaluate an algorithm on some interesting problem of your choice;
- Theory and Algorithms: study theoretically or empirically some new machine learning algorithm/problem.
Previous project suggestions: Spring 2012 projects
Follow the link for each class to find a detailed description, suggested readings, and class slides. Some of the later classes may be subject to reordering or rescheduling.
Class Date Title Instructor(s) Class 01 Wed 04 Sep The Course at a Glance TP Class 02 Mon 09 Sep The Learning Problem and Regularization TP Class 03 Wed 11 Sep Reproducing Kernel Hilbert Spaces LR Class 04 Mon 16 Sep Mercer Theorem and Feature Maps LR Class 05 Wed 18 Sep Tikhonov Regularization and the Representer Theorem LR Class 06 Mon 23 Sep Regularized Least Squares and Support Vector Machines LR Class 07 Wed 25 Sep Generalization Bounds, Intro to Stability LR/TP Class 08 Mon 30 Sep Stability of Tikhonov Regularization LR/TP Class 09 Wed 2 Oct Regularization Parameter Choice: Theory and Practice LR Class 10 Mon 07 Oct Bayesian Interpretations of Regularization LR Class 11 Wed 09 Oct Spectral Regularization LR Monday 14 October - Columbus Day Class 12 Wed 16 Oct Regularization for Multi-Output Learning LR Class 13 Thurs 17 Oct Loose ends, Project discussions Class 14 Mon 21 Oct Sparsity Based Regularization 1 LR Class 15 Wed 23 Oct Sparsity Based Regularization 2 LR Class 16 Mon 28 Oct Regularization with Multiple Kernels LR Class 17 Wed 30 Oct On-line Learning LR Class 18 Mon 04 Nov Manifold Regularization LR Class 19 Wed 06 Nov Hierarchical Representation for Learning: Visual Cortex TP Monday 11 November - Veterans Day Class 20 Tues 12 Nov Hierarchical Representation for Learning: Mathematics LR Class 21 Wed 13 Nov Hierarchical Representation for Learning: Computational Model TP Class 22 Mon 18 Nov Learning Data Representation with Regularization LR Class 23 Wed 20 Nov TBA Class 24 Mon 25 Nov TBA Class 25 Wed 27 Nov Project Presentations Class 26 Mon 02 Dec Project Presentations
Reading ListThere is no textbook for this course. All the required information will be presented in the slides associated with each class. The books/papers listed below are useful general reference reading, especially from the theoretical viewpoint. A list of suggested readings will also be provided separately for each class.
- Bousquet, O., S. Boucheron and G. Lugosi. Introduction to Statistical Learning Theory. Advanced Lectures on Machine Learning Lecture Notes in Artificial Intelligence 3176, 169-207. (Eds.) Bousquet, O., U. von Luxburg and G. Ratsch, Springer, Heidelberg, Germany (2004)
- F. Cucker and S. Smale. On The Mathematical Foundations of Learning. Bulletin of the American Mathematical Society, 2002.
- F. Cucker and D-X. Zhou. Learning theory: an approximation theory viewpoint. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, Cambridge, 2007.
- L. Devroye, L. Gyorfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer, 1997.
- T. Evgeniou and M. Pontil and T. Poggio. Regularization Networks and Support Vector Machines. Advances in Computational Mathematics, 2000.
- T. Poggio and S. Smale. The Mathematics of Learning: Dealing with Data. Notices of the AMS, 2003
- I. Steinwart and A. Christmann. Support vector machines. Springer, New York, 2008.
- V. N. Vapnik. Statistical Learning Theory. Wiley, 1998.
- V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
- N. Cristianini and J. Shawe-Taylor. Introduction To Support Vector Machines. Cambridge, 2000.
Background Mathematics References
- A. N. Kolmogorov and S. V. Fomin, Introductory Real Analysis, Dover Publications, 1975.
- A. N. Kolmogorov and S. V. Fomin, Elements of the Theory of Functions and Functional Analysis, Dover Publications, 1999.
- Luenberger, Optimization by Vector Space Methods, Wiley, 1969.
Neuroscience Related References
- Serre, T., L. Wolf, S. Bileschi, M. Riesenhuber and T. Poggio. "Object Recognition with Cortex-like Mechanisms", IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 3, 411-426, 2007.
- Serre, T., A. Oliva and T. Poggio."A Feedforward Architecture Accounts for Rapid Categorization", Proceedings of the National Academy of Sciences (PNAS), Vol. 104, No. 15, 6424-6429, 2007.
- S. Smale, L. Rosasco, J. Bouvrie, A. Caponnetto, and T. Poggio. "Mathematics of the Neural Response", Foundations of Computational Mathematics, Vol. 10, 1, 67-91, June 2009.