Liu Ziyin

Liu Ziyin

Email: liu.ziyin.p (at) gmail.com / ziyinl (at) mit.edu
Office: Room 26-209, MIT



I am a Postdoctoral Fellow at MIT and NTT Research. At MIT, I work with Prof. Isaac Chuang. I also collaborate with Prof. Tomaso Poggio in the BCS department. My research focus is on the theoretical foundation of deep learning. Prior to coming to MIT, I received my PhD in physics at the University of Tokyo under the supervision of Prof. Masahito Ueda. I received a Bachelor's degree in physics and mathematics at Carnegie Mellon University. I also serve as an Area Chair for NeurIPS. Personally, I am interested in art, literature, and philosophy. I also play Go. Also, Paul Liang is my great great great friend. If you have questions or want to collaborate, or just want to say hi, feel free to send an email.

A talk I gave recently: How does physics help understand deep learning?

Doctor thesis: Symmetry breaking in deep learning (深層学習に於ける対称性の破れ, 2023).
Master thesis: Mean-field learning dynamics of deep neural networks (2020).

Research Interest

I am particularly interested identifying scientific principles of artificial intelligence (what is a principle?), and I think tools and intuitions from other fields of sciences an be of great help. Broadly speaking, I work to advance the following fields

NTT interns I work(ed) with

Recent Preprints

  1. Neural Thermodynamics I: Entropic Forces in Deep and Universal Representation Learning
    2. Liu Ziyin*, Yizhou Xu*, Isaac Chuang
    Preprint 2025
    [arXiv]
  2. Heterosynaptic Circuits Are Universal Gradient Machines
    3. Liu Ziyin, Isaac Chuang, Tomaso Poggio
    Preprint 2025
    [arXiv]
  3. Parameter Symmetry Breaking and Restoration Determines the Hierarchical Learning in AI Systems
    4. Liu Ziyin, Yizhou Xu, Tomaso Poggio, Isaac Chuang
    Preprint 2025
    [arXiv]
  4. Self-Assembly of a Biologically Plausible Learning Circuit
    5. Qianli Liao*, Liu Ziyin*, Yulu Gan*, Brian Cheung, Mark Harnett, Tomaso Poggio
    Preprint 2024
    [arXiv]

Selected Work

  1. Formation of Representations in Neural Networks
    2. Liu Ziyin, Isaac Chuang, Tomer Galanti, Tomaso Poggio
    ICLR 2025 (spotlight: 5% of all submissions)
    [paper]
  2. Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent
    3. Liu Ziyin, Mingze Wang, Hongchao Li, Lei Wu
    NeurIPS 2024
    [paper] [arXiv]
  3. Symmetry Induces Structure and Constraint of Learning
    4. Liu Ziyin
    ICML 2024
    [arXiv]
  4. Sparsity by Redundancy: Solving L1 with SGD
    5. Liu Ziyin*, Zihao Wang*
    ICML 2023
    [paper] [arXiv]
  5. Exact Solutions of a Deep Linear Network
    6. Liu Ziyin, Botao Li, Xiangming Meng
    NeurIPS 2022
    [paper] [arXiv]
  6. SGD Can Converge to Local Maxima
    7. Liu Ziyin, Botao Li, James B. Simon, Masahito Ueda
    ICLR 2022 (spotlight: 5% of all submissions)
    [paper] [arXiv]
  7. Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
    8. Kangqiao Liu*, Liu Ziyin*, Masahito Ueda
    ICML 2021
    [paper] [arXiv]
  8. Neural Networks Fail to Learn Periodic Functions and How to Fix It
    9. Liu Ziyin, Tilman Hartwig, Masahito Ueda
    NeurIPS 2020
    [paper] [arXiv]
  9. Deep Gamblers: Learning to Abstain with Portfolio Theory
    10. Liu Ziyin, Zhikang Wang, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency, Masahito Ueda
    NeuRIPS 2019
    [paper] [arXiv] [code]

Publications

  1. Law of Balance and Stationary Distribution of Stochastic Gradient Descent
    2. Liu Ziyin*, Hongchao Li*, Masahito Ueda
    Physical Review E
    [arXiv]
  2. Compositional Generalization Requires More Than Disentangled Representations
    3. Qiyao Liang, Daoyuan Qian, Liu Ziyin, Ila Fiete
    ICML 2025
    [arXiv]
  3. Understanding the Emergence of Multimodal Representation Alignment
    4. Megan Tjandrasuwita, Chanakya Ekbote, Liu Ziyin, Paul Pu Liang
    ICML 2025
    [arXiv]
  4. Formation of Representations in Neural Networks
    5. Liu Ziyin, Isaac Chuang, Tomer Galanti, Tomaso Poggio
    ICLR 2025 (spotlight: 5% of all submissions)
    [paper]
  5. Remove Symmetries to Control Model Expressivity
    6. Liu Ziyin*, Yizhou Xu*, Isaac Chuang
    ICLR 2025
    [paper]
  6. When Does Feature Learning Happen? Perspective from an Analytically Solvable Model
    7. Yizhou Xu*, Liu Ziyin*
    ICLR 2025
    [paper]
  7. Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent
    8. Liu Ziyin, Mingze Wang, Hongchao Li, Lei Wu
    NeurIPS 2024
    [paper] [arXiv]
  8. Symmetry Induces Structure and Constraint of Learning
    9. Liu Ziyin
    ICML 2024
    [arXiv]
  9. Zeroth, first, and second-order phase transitions in deep neural networks
    10. Liu Ziyin, Masahito Ueda
    Physical Review Research 2023
    [arXiv]
  10. Exact Solutions of a Deep Linear Network
    11. Liu Ziyin, Botao Li, Xiangming Meng
    Journal of Statistical Mechanics: Theory and Experiment, 2023
    [paper] [arXiv]
  11. On the stepwise nature of self-supervised learning
    12. James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, Joshua Albrecht
    ICML 2023
    [paper] [arXiv]
  12. Sparsity by Redundancy: Solving L1 with SGD
    13. Liu Ziyin*, Zihao Wang*
    ICML 2023
    [paper] [arXiv]
  13. What shapes the loss landscape of self-supervised learning?
    14. Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda, Hidenori Tanaka
    ICLR 2023
    [paper] [arXiv]
  14. Exact Solutions of a Deep Linear Network
    15. Liu Ziyin, Botao Li, Xiangming Meng
    NeurIPS 2022
    [paper] [arXiv]
  15. Posterior Collapse of a Linear Latent Variable Model
    16. Zihao Wang*, Liu Ziyin*
    NeurIPS 2022 (oral: 1% of all submissions)
    [paper] [arXiv]
  16. Universal Thermodynamic Uncertainty Relation in Non-Equilibrium Dynamics
    17. Liu Ziyin, Masahito Ueda
    Physical Review Research (2022)
    [paper] [arXiv]
  17. Theoretically Motivated Data Augmentation and Regularization for Portfolio Construction
    18. Liu Ziyin, Kentaro Minami, Kentaro Imajo
    ICAIF 2022 (3rd ACM International Conference on AI in Finance)
    [paper] [arXiv]
  18. Power Laws and Symmetries in a Minimal Model of Financial Market Economy
    19. Liu Ziyin, Katsuya Ito, Kentaro Imajo, Kentaro Minami
    Physical Review Research (2022)
    [paper] [arXiv]
  19. Logarithmic landscape and power-law escape rate of SGD
    20. Takashi Mori, Liu Ziyin, Kangqiao Liu, Masahito Ueda
    ICML 2022
    [paper] [arXiv]
  20. SGD with a Constant Large Learning Rate Can Converge to Local Maxima
    21. Liu Ziyin, Botao Li, James B. Simon, Masahito Ueda
    ICLR 2022 (spotlight: 5% of all submissions)
    [paper] [arXiv]
  21. Strength of Minibatch Noise in SGD
    22. Liu Ziyin*, Kangqiao Liu*, Takashi Mori, Masahito Ueda
    ICLR 2022 (spotlight: 5% of all submissions)
    [paper] [arXiv]
  22. On the Distributional Properties of Adaptive Gradients
    23. Zhang Zhiyi*, Liu Ziyin*
    UAI 2021
    [paper] [arXiv]
  23. Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
    24. Kangqiao Liu*, Liu Ziyin*, Masahito Ueda
    ICML 2021
    [paper] [arXiv]
  24. Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
    25. Paul Pu Liang*, Peter Wu*, Liu Ziyin, Louis-Philippe Morency, Ruslan Salakhutdinov
    ACM Multimedia 2021
    NeurIPS 2020 Workshop on Meta Learning
    [arXiv] [code]
  25. Neural Networks Fail to Learn Periodic Functions and How to Fix It
    26. Liu Ziyin, Tilman Hartwig, Masahito Ueda
    NeurIPS 2020
    [paper] [arXiv]
  26. Deep Gamblers: Learning to Abstain with Portfolio Theory
    27. Liu Ziyin, Zhikang Wang, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency, Masahito Ueda
    NeuRIPS 2019
    [paper] [arXiv] [code]
  27. Think Locally, Act Globally: Federated Learning with Local and Global Representations
    28. Paul Pu Liang*, Terrance Liu*, Liu Ziyin, Ruslan Salakhutdinov, Louis-Philippe Morency
    NeurIPS 2019 Workshop on Federated Learning (oral, distinguished student paper award)
    [paper] [arXiv] [code]
  28. Multimodal Language Analysis with Recurrent Multistage Fusion
    29. Paul Pu Liang, Ziyin Liu, Amir Zadeh, Louis-Philippe Morency
    EMNLP 2018 (oral presentation)
    [paper] [supp] [arXiv] [slides]
This page has been accessed several times since July 07, 2018.