Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension

Bounds on the smallest eigenvalue of the neural tangent kernel (NTK) are a key ingredient in the analysis of neural network optimization and memorization. However, existing results require distributional assumptions on the data and are limited to a high-dimensional setting, where the input dimension d0 scales at least logarithmically in the number of samples n. In this work we remove both of these requirements and instead provide bounds in terms of a measure of the collinearity of the data: notably these bounds hold with high probability even when d0 is held constant versus n. We prove our results through a novel application of the hemisphere transform. This is work with Kedar Karhadkar and Michael Murray.

Bio
https://www.math.ucla.edu/~montufar/

Guido Montúfar is a Professor of Mathematics and Statistics & Data Science at UCLA. He is also the leader of the Math Machine Learning Group at the Max Planck Institute for Mathematics in the Sciences. His research focuses on the mathematical foundations of machine learning and specifically deep learning theory. He studied mathematics and theoretical physics at TU Berlin and obtained the Dr.rer.nat. in Mathematics in 2012 as an IMPRS fellow in Leipzig. His work has been recognized with awards from the ERC, DFG, NSF. Guido Montufar is a 2022 Alfred P. Sloan Research Fellow.