Energetic Variational Inference

Variational Inference (VI) is an important research area in the field of machine learning. Many VI approaches have been developed and widely used in machine learning and other related areas. In this talk, we propose a new VI framework, called energetic variational inference, or EVI. It minimizes the VI objective function based on a prescribed energy-dissipation law. Under the EVI, we can recover many existing Particle-based Variational Inference (ParVI) methods, including the popular Stein variational gradient descent (SVGD). More importantly, many new ParVI schemes can be created under this framework. To demonstrate how to develop a new EVI method, we propose a new particle-based EVI scheme, which performs the particle-based approximation of the density first and then uses the approximated density in the variational procedure, or "Approximation-then-Variation'' for short. Thanks to this order of approximation and variation, the new scheme can maintain the variational structure at the particle level. Different divergence measures can be combined with EVI to produce different ParVI algorithms. Specially, we demonstrate the EVI methods using KL-divergence and Maximum Mean Discrepancy measure. The proposed methods are compared with existing ones using different examples. We conclude the talk with discussions of future research topics.

About the speaker

Dr. Lulu Kang is an Associate Professor of the Department of Applied Math at Illinois Institute of Technology. She holds an M.S. in Operations Research and a Ph.D. in Industrial Engineering from the Stewart School of Industrial and Systems Engineering at Georgia Institute of Technology. Dr. Kang has worked on various areas in related to Statistics, including uncertainty quantification, statistical design and analysis of experiments, Bayesian computational statistics, etc. She has publications and submitted papers in top statistical journals including Techometrics, SIAM/ASA Journal on Uncertainty Quantification, Statistica Sinica, etc. Dr. Kang is currently the associate editor for journals SIAM/ASA Journal on Uncertainty Quantification and Technometrics.