Kenji Kawaguchi

Massachusetts Institute of Technology (MIT)
Computer Science and Artificial Intelligence Laboratory (CSAIL)


Selected Publications

Journal Articles

Ameya D. Jagtap, Kenji Kawaguchi, George E. Karniadakis. Adaptive Activation Functions Accelerate Convergence in Deep and Physics-informed Neural Networks. Journal of Computational Physics, 404, 109136, 2020.
[pdf] [BibTex]

Kenji Kawaguchi and Yoshua Bengio. Depth with Nonlinearity Creates No Bad Local Minima in ResNets. Neural Networks, 118, 167-174, 2019.
[pdf] [BibTex] [Video] One of the 25 most downloaded articles from Neural Networks in the last 90 days (link)

Kenji Kawaguchi, Jiaoyang Huang and Leslie Pack Kaelbling. Every Local Minimum Value is the Global Minimum Value of Induced Model in Non-convex Machine Learning. Neural Computation, 31(12), 2293-2323, MIT press, 2019.
[pdf] [BibTex]

Kenji Kawaguchi, Jiaoyang Huang and Leslie Pack Kaelbling. Effect of Depth and Width on Local Minima in Deep Learning. Neural Computation, 31(7), 1462-1498, MIT press, 2019.
[pdf] [BibTex]

Kenji Kawaguchi, Yu Maruyama and Xiaoyu Zheng. Global Continuous Optimization with Error Bound and Fast Convergence. Journal of Artificial Intelligence Research (JAIR), 56, 153-195, 2016.
[pdf] [BibTex]

Xiaoyu Zheng, Hiroto Itoh, Kenji Kawaguchi, Hitoshi Tamaki and Yu Maruyama. Application of Bayesian nonparametric models to the uncertainty and sensitivity analysis of source term in a BWR severe accident. Reliability Engineering and System Safety (RESS), 138, 253-262, 2015.
[pdf] [BibTeX]

Jun Ishikawa, Kenji Kawaguchi and Yu Maruyama. Analysis for iodine release from unit 3 of Fukushima Dai-ichi nuclear power plant with consideration of water phase iodine chemistry. Journal of Nuclear Science and Technology (JNST), 52(3), 308-315, 2015.
[pdf] [BibTeX]

Conference Papers

Kenji Kawaguchi and Leslie Pack Kaelbling. Elimination of All Bad Local Minima in Deep Learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), accepted, 2020.
[pdf] [BibTeX] [Video]

Kenji Kawaguchi* and Haihao Lu*. Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimization. In International Conference on Artificial Intelligence and Statistics (AISTATS), accepted, 2020.
[pdf] [BibTeX] [Code]

Kenji Kawaguchi and Jiaoyang Huang. Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizes. In Proceedings of the 57th Allerton Conference on Communication, Control, and Computing (Allerton), 2019.
[pdf] [BibTex] [Video]

Kenji Kawaguchi*, Bo Xie*, Vikas Verma, and Le Song. Deep Semi-Random Features for Nonlinear Function Approximation. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI), 2018.
[pdf] [BibTex]

Kenji Kawaguchi. Deep Learning without Poor Local Minima. In Advances in Neural Information Processing (NeurIPS), 2016.
[pdf] [BibTex] [Spotlight Video] [Talk] Selected for NeurIPS oral presentation (2% submissions)

Kenji Kawaguchi. Bounded Optimal Exploration in MDP. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI), 2016.
[pdf] [BibTex]

Kenji Kawaguchi, Leslie Pack Kaelbling and Tomás Lozano-Pérez. Bayesian Optimization with Exponential Convergence. In Advances in Neural Information Processing (NeurIPS), 2015.
[pdf] [BibTex] [Code]

Book Chapter

Kenji Kawaguchi, Leslie Pack Kaelbling and Yoshua Bengio. Generalization in Deep Learning. In Mathematics of Deep Learning, Cambridge University Press, to appear. Prepint available as: MIT-CSAIL-TR-2018-014, Massachusetts Institute of Technology, 2018.
[pdf] [BibTex] [Code]

Technical Reports

Tomaso Poggio, Kenji Kawaguchi, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Xavier Boix, Jack Hidary and Hrushikesh Mhaskar. Theory of Deep Learning III: explaining the non-overfitting puzzle. Massachusetts Institute of Technology CBMM Memo No. 73, 2018.
[pdf] [BibTex]

Qianli Liao, Kenji Kawaguchi and Tomaso Poggio. Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning. Massachusetts Institute of Technology CBMM Memo No. 57, 2016.
[pdf] [BibTeX]

*Equal contribution


Invited talks

Invited talk at International Conference:
Minisymposium on Theoretical Foundations of Deep Learning, ICIAM 2019, Spain.

Seminar, university, and research lab:
• Brown University / Seminar, 2020.
• University of Michigan, Ann Arbor / Seminar, 2020.
• National University of Singapore / Seminar, 2020.
• University of British Columbia / Seminar, 2020.
• Stanford University / CS theory lunch, 2019.
• Harvard University / Professor Horng-Tzer Yau lab, 2019.
• Carnegie Mellon University (CMU) / AI Seminar Series, 2019.
• Carnegie Mellon University (CMU) / Professor Eric P. Xing lab, 2019.
• Toyota Technological Institute at Chicago (TTIC) / Young Researcher Seminar Series, 2019.
• Purdue University / Seminar at School of Industrial Engineering, 2019.
• PhILMs center / invited by Professor George Em Karniadakis at Brown University, 2019.
• Google Research (at Cambridge) / invited by Dr. Dilip Krishnan, 2017.
• MIT / Professor David Sontag lab, 2017.
• MIT / Professor Tomaso Poggio lab, 2016.
• MIT / Machine Learning Tea, 2016.


Selected Professional Service and Activity

Invited research visits:
• Microsoft Research (MSR), Redmond, Summer 2018.
• TTIC, Chicago, Fall 2019.
• University of Cambridge: Invited to be in the participant list of the 2021 program on "Mathematics of Deep Learning" by the organization team of Prof. Peter Bartlett, Prof. Arnulf Jentzen, Prof. Anders Hansen, Prof. Gitta Kutyniok, Prof. Stephane Mallat, and Prof. Carola Schönlieb.

Program Committee Member:
• AAAI Conference on Artificial Intelligence (AAAI), 2020 and 2019.
• Conference on Uncertainty in Artificial Intelligence (UAI), 2019.

Invited Journal reviewer:
• Journal of Machine Learning Research (JMLR)
• Annals of Statistics (Ann. Stat.)
• Neural Computation (MIT press)
• Neural Networks (Elsevier)
• IEEE Transactions on Neural Networks and Learning Systems (IEEE TNNLS)

Invited Conference reviewer:
• NeurIPS 2020, ICML 2020, NeurIPS 2019

Selected poster presentations:
• Institute for Advanced Study (IAS): Workshop on Theory of Deep Learning: Where next?, organized by Professors Sanjeev Arora, Joan Bruna, Rong Ge, Suriya Gunasekar, Jason Lee, Bin Yu, 2019.
• CRM/CIFAR Deep Learning Summer School, organized by Professors Aaron Courville and Yoshua Bengio, 2016.


Education

Massachusetts Institute of Technology              present
Ph.D. student, Electrical Engineering and Computer Science
Advisor: Leslie Pack Kaelbling
Thesis committee: Yoshua Bengio and Suvrit Sra
GPA 5.0/5.0 (+ highest internal grades: all A+ except one class where A was the highest offered)

Massachusetts Institute of Technology             2016/02
S.M., Electrical Engineering and Computer Science
Advisors: Leslie Pack Kaelbling and Tomás Lozano-Pérez.
Thesis: Towards Practical Theory: Bayesian Optimization and Optimal Exploration
GPA 5.0/5.0 (+ highest internal grades)


Code

Bayesian optimization with exponential convergence

Generalization in Deep Learning

Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimization