We will present four talks around two cardinal aspects: (1) introducing tools common in RMT that can be applied to machine learning, and (2) Recent applications of RMT in optimization, generalization, and statistical learning theory. The tutorial is divided into 4 parts of roughly 30 min each.
This section introduces two of the classical Random Matrix Theory ensembles: the Gaussian Orthogonal Ensemble and Wishart matrices. We’ll see the emergence of the semi-circle and Marchenko-Pastur distributions from simple experiments on these matrices. More …
This section introduces some two very powerful objects in random matrix theory: the Stieltjes and R transform. We’ll see how to use them to construct increasingly complex ensembles and discuss some recent topics in random matrix theory such as freeness. More …
This section is devoted to applications of random matrix theory into the analysis of numerical algorithms. We’ll see how random matrix theory allows to avoid the pitfals of a worst-case analysis and why the time it takes for an algorithm to return a solution seems to be only mildly dependent on the data distribution over the inputs, a phenomenon known as halting time universality. More …
This section discusses random matrix theory models of generalization in Deep Neural Networks. We’ll see how these models can be used to study some of the most puzzling behaviors of deep neural networks, such as double (and triple) descent. More …
This section introduces two of the classical Random Matrix Theory ensembles, the Gaussian Orthogonal Ensembleand Wishart matrices. Through numerical experiments, we’ll motivate some of the most important distributions in random matrix theory such as the semi-circle and Marchenko-Pastur, as well as key concepts such as universality.
Slides PDF | Slides source code | Experiments (Colab) | Video with Slides
This section introduces some core proof techniques in random matrix theory: the Stieltjes and R transform. Through experiments, we will illustrate how to use these objects to compute spectral properties of increasingly complex random matrix ensembles. We’ll also discuss some recent topics in random matrix theory such as freeness.
Slides PDF | Slides source code | Experiments R-Transform (by Ria Stevens) | Experiments Stieltjes Transform (by Elliot Paquette) | Video with Slides
This section is devoted to applications of random matrix theory into the analysis of numerical algorithms. We’ll see how random matrix theory allows to avoid the pitfals of a worst-case analysis and why the time it takes for an algorithm to return a solution seems to be only mildly dependent on the data distribution over the inputs, a phenomenon known as halting time universality.
Slides PDF | Slides source code | Experiments (Colab) | Video with Slides
This section discusses random matrix theory models of generalization in Deep Neural Networks. We’ll see how these models can be used to study some of the most puzzling behaviors of deep neural networks, such as double (and triple) descent.
This tutorial is meant as an introduction to the field of random matrix theory for machine learning researchers. Those wanting to deepen their understanding might be interested in the following references.
Textbooks on random matrix theory:
Textbook on random matrix theory for machine learning:
The full list of references used in the slides can be found here.
The researchers that will guide you through this tutorial are
Fabian Pedregosa’s research has focused in the last years in developing an average-case analysis of optimization algorithms using techniques from random matrix theory. Previously, he has done research in projection-free and asynchronous optimizationbeen. He is also one of the founding members of the scikit-learn machine learning library. Webpage.
Courtney Paquette’s research broadly focuses on designing and analyzing algorithms for large-scale optimization problems, motivated by applications in data science. Her recent work has focused on average-case complexity that combined aspects of optimization and random matrix theory. She is a CIFAR Canada AI chair with the Quebec AI institute (MILA).
Tom Trogdon’s work focuses on connections between classical algorithms from numerical linear algebra and random matrix theory. He is a core member of a group of researchers that have, first, discovered that the runtimes of algorithms exhibit universal distributions, and second, began rigorously establishing theorems.
Jeffrey Pennington’s current research interests center on the theory of deep learning, and include topics such as random matrix theory and generalization in high dimensions, the role of overparameterization, wide neural networks and their corresponding kernels, the dynamics of learning, and the geometry of high-dimensional loss surfaces.