AI for Science and Science for AI
Artificial Intelligence techniques have progressed immensely over the past few years:
many fields of science and engineering can now make use of these techniques.
We are very interested in exploring how AI can help our research, and science
research in general, but we are also quite interested how
physics insights can lead to even more powerful AI algorithms. At this web
page, you can learn more about this part of our research.
Applications of AI techniques in science and engineering
AI techniques enabled amazing applications in computer vision, game playing, question
answering, etc. AI can and will have similar dramatic impact on development of
science and engineering. In order to achieve this, sometimes the existing AI
techniques (developed for applications other than science) can be used as they are.
But, more often, they need to be modified to be suitable for science applications,
sometimes a little, sometimes a lot. Sometimes, completely new AI techniques need to
be discovered for certain science applications. For some examples of our work on
this, please see:
-
"TENG: Time-Evolving Natural Gradient for Solving PDEs With Deep Neural Nets Toward Machine Precision"
Zhuo Chen, Jacob Mccarran, Esteban Vizcaino, Marin Soljacic, Di Luo.
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:7143-7162 (2024).
-
"QuACK: Accelerating Gradient-Based Quantum Optimization with Koopman Operator Learning"
Di Luo, Jiayu Shen, Rumen Dangovski, Marin Soljacic.
NeurIPS (2023).
-
"ANTN: Bridging Autoregressive Neural Networks and Tensor Networks for Quantum Many-Body Simulation"
Zhuo Chen, Laker Newhouse, Eddie Chen, Di Luo, Marin Soljacic.
NeurIPS (2023).
-
"Q-Flow: Generative Modeling for Differential Equations of Open Quantum Dynamics with Normalizing Flows"
(pdf)
Owen M Dugan, Peter Y. Lu, Rumen Dangovski, Di Luo, Marin Soljacic.
Proceedings of the 40th International Conference on Machine Learning, PMLR 202, 2023.
-
"Deep Learning for Bayesian Optimization of Scientific Problems with High-Dimensional Structure"
Samuel Kim, Peter Y. Lu, Charlotte Loh, Jamie Smith, Jasper Snoek, Marin Soljacic.
Transactions on Machine Learning Research, (2022).
-
"Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science"
Charlotte Loh, Thomas Christensen, Rumen Dangovski, Samuel Kim & Marin Soljacic.
Nature Communications Vol.13, Article number: 4223 (2022).
-
"Predictive and generative machine learning models for photonic crystals"
Thomas Christensen, Charlotte Loh, Stjepan Picek, Domagoj Jakobovic, Li Jing, Sophie Fisher, Vladimir Ceperic, John D. Joannopoulos and Marin Soljacic.
Invited paper in Nanophotonics Vol.9, p.4183, (2020).
-
"Migrating Knowledge between Physical Scenarios based on Artificial Neural Networks"
(pdf)
Yurui Qu, Li Jing, Yichen Shen, Min Qiu, and Marin Soljacic.
ACS Photonics 2019 Vol.6, p.1168.
-
"Nanophotonic particle simulation and inverse design using artificial neural networks"
(pdf)
John Peurifoy, Yichen Shen, Li Jing, Yi Yang, Fidel Cano-Renteria, Brendan Delacy, Max Tegmark, John D. Joannopoulos, Marin Soljacic.
Science Advances, Vol.4, no.6, eaar4206, (2018).
More material
(here).
Interpretable AI
While today's AI algorithms work amazingly well, most of them operate in
quite a "black box" fashion: they provide (often) correct answers, but it is
close to impossible to understand how they work or reach answers. For many
applications, it would be highly desirable to have AI algorithms that not only
provide correct answers but also it is easy to understand how these answers
were obtained. For most applications, that would give us more faith into
these algorithms, and more transparency when they can be fully trusted, and when not.
Moreover, in science, such algorithms could also provide physical insights,
and be useful in developing new science theories.
For some of our work on interpretable AI, see:
-
"Deep Learning and Symbolic Regression for Discovering Parametric Equations"
(pdf)
Michael Zhang, Samuel Kim, Peter Y. Lu, Marin Soljacic.
IEEE Transactions on Neural Networks and Learning Systems, DOI: 10.1109/TNNLS.2023.3297978, (2023).
-
"Discovering conservation laws using optimal transport and manifold learning"
(pdf)
Peter Y. Lu, Rumen Dangovski, Marin Soljacic.
Nature Communications, Vol.14, Article number: 4744 (2023).
-
"Topogivity: A Machine-Learned Chemical Rule for Discovering Topological Materials"
(pdf)
Andrew Ma, Yang Zhang, Thomas Christensen, Hoi Chun Po, Li Jing, Liang Fu, and Marin Soljacic.
Nano Lett. Vol.23, p.772, (2023).
-
"Discovering sparse interpretable dynamics from partial observations"
Peter Y. Lu, Joan Arino Bernad & Marin Soljacic.
Communications Physics Vol.5, Article number: 206, (2022).
-
"Extracting Interpretable Physical Parameters from Spatiotemporal Systems Using Unsupervised Learning"
Peter Y. Lu, Samuel Kim, and Marin Soljacic.
Phys. Rev. X Vol.10, p.031056, (2020).
More material (here).
-
"Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery"
Samuel Kim, Peter Y. Lu, Srijon Mukherjee, Michael Gilbert, Li Jing, Vladimir Ceperic, Marin Soljacic.
IEEE Transactions on Neural Networks and Learning Systems DOI: 10.1109/TNNLS.2020.3017010 (2020).
New Hardware for AI
Almost all computing nowadays is done using electrons. However, there are a few known algorithms that can be implemented in a superior way using light (photons). For example, matrix multiplication can be implemented with light essentially instantly (at light-speed), and theoretically with zero energy consumption. Since deep learning algorithms rely so heavily on matrix multiplication, there is a value proposition for implementing some of them with photonic (instead of electronic) hardware. Such systems
(Nature Photonics 2017)
could for certain applications be substantially faster, consume much less energy, and have much lower latency.
For our additional work on photonics hardware, see:
-
"Photonic Probabilistic Machine Learning Using Quantum Vacuum Noise"
(pdf)
Seou Choi, Yannick Salamin, Charles Roques-Carmes, Rumen Dangovski, Di Luo, Zhuo Chen, Michael Horodynski, Jamison Sloan, Shiekh Zia Uddin, and Marin Soljacic.
Nature Communications, Vol.15, p.7760 (2024).
-
"Biasing the quantum vacuum to control macroscopic probability distributions"
(pdf)
Charles Roques-Carmes, Yannick Salamin, Jamison Sloan, Seou Choi, Gustavo Velez, Ethan Koskas, Nicholas Rivera, Steven E. Kooi, John D. Joannopoulos, Marin Soljacic.
Science Vol.381, p.205, (2023).
More material
(here).
-
"Inference in artificial intelligence with deep optics and photonics" Gordon Wetzstein, Aydogan Ozcan, Sylvain Gigan, Shanhui Fan, Dirk Englund, Marin Soljacic, Cornelia Denz, David A. B. Miller, and Demetri Psaltis. Nature Vol.588, p.39 (2020).
-
"Accelerating recurrent Ising machines in photonic integrated circuits" Mihika Prabhu, Charles Roques-Carmes, Yichen Shen, Nicholas Harris, Li Jing, Jacques Carolan, Ryan Hamerly, Tom Baehr-Jones, Michael Hochberg, Vladimir Ceperic, John D. Joannopoulos, Dirk R. Englund, and Marin Soljacic. Optica Vol.7, p.551, (2020).
-
"Heuristic recurrent algorithms for photonic Ising machines" Charles Roques-Carmes, Yichen Shen, Cristian Zanoci, Mihika Prabhu, Fadi Atieh, Li Jing, Tena Dubcek, Chenkai Mao, Miles R. Johnson, Vladimir Ceperic, John D. Joannopoulos, Dirk Englund, Marin Soljacic. Nature Communications Vol.11, Article number: 249 (2020).
New Algorithms for AI
Our group's strong background in general physics and math techniques also enables us to sometimes use these techniques to understand and analyze various AI algorithms, or construct novel AI algorithms with improved performance. On this topic, we often collaborate with our colleagues from computer science departments and/or industry. Some of examples of our work on this are described below.
Using unitary (instead of general) matrices in artificial neural networks (ANNs) is a promising way to solve the gradient explosion/vanishing problem, as well as to enable ANNs to learn long-term correlations in the data. We present (ICML 2017, with Yann LeCun's group) an Efficient Unitary Neural Network (EUNN) architecture that parametrizes the entire space of unitary matrices in a complete and computationally efficient way, thereby eliminating the need for time-consuming unitary subspace-projections.
In another project, we present (Neural Computation 2019, with Yoshua Bengio's group) a novel recurrent neural network (RNN) based model that combines the remembering ability of unitary RNNs with the ability of gated RNNs to effectively forget redundant/irrelevant information in its memory. We achieve this by extending unitary RNNs with a gating mechanism.
The concepts of unitary evolution matrices and associative memory have boosted the field of Recurrent Neural Networks (RNN) to state-of-the-art performance in a variety of sequential tasks. However, RNN still have a limited capacity to manipulate long-term memory. To bypass this weakness the most successful applications of RNN use external techniques such as attention mechanisms. In yet another project, (TACL 2019) we propose a novel RNN model that unifies the state-of-the-art approaches: Rotational Unit of Memory (RUM). The core of RUM is its rotational operation, which is, naturally, a unitary matrix, providing architectures with the power to learn long-term dependencies by overcoming the vanishing and exploding gradients problem. Moreover, the rotational unit also serves as associative memory. Our work with RUM turns out to be particularly suitable for Natural Language Processing (NLP), so we applied it to the task of summarizing scientific articles. In a further work, (AAAI 2021, with Preslav Nakov's group) we also explored transformers for the same summarization task.
In a next project, (ICLR 2022 with Pulkit Agrawal's group) we have shown that pre-training that encourages non-trivial equivariance to some transformations, while maintaining invariance to other transformations, can be used to improve the semantic quality of representations.
For our additional work on AI algorithms, see:
-
"OccamLLM: Fast and Exact Language Model Arithmetic in a Single Step"
(pdf)
Owen Dugan, Donato Manuel Jimenez Beneto, Charlotte Loh, Zhuo Chen, Rumen Dangovski, and Marin Soljacic.
NeurIPS (2024).
-
"QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation"
(pdf)
Zhuo Chen, Rumen Dangovski, Charlotte Loh, Owen Dugan, Di Luo, Marin Soljacic.
NeurIPS (2024).
-
"Mitigating Confirmation Bias in Semi-supervised Learning via Efficient Bayesian Model Averaging"
Charlotte Loh, Rumen Dangovski, Shivchander Sudalairaj, Seungwook Han, Ligong Han, Leonid Karlinsky, Marin Soljacic, Akash Srivastava.
Transactions on Machine Learning Research (2023).
-
"Learning to Optimize Quasi-Newton Methods"
Isaac Liao, Rumen Dangovski, Jakob Nicolaus Foerster, Marin Soljacic.
Transactions on Machine Learning Research (2023).
-
"Multi-Symmetry Ensembles: Improving Diversity and Generalization via Opposing Symmetries"
(pdf)
Charlotte Loh, Seungwook Han, Shivchander Sudalairaj, Rumen Dangovski, Kai Xu, Florian Wenzel, Marin Soljacic, Akash Srivastava.
Proceedings of the 40th International Conference on Machine Learning, Honolulu, Hawaii, USA. PMLR 202, 2023.
-
"DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings"
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, James Glass.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics, p. 4207 (2022).