Stable Training of Normalizing Flows for Variational Inference

Variational inference with Normalizing Flows (NFs) is an increasingly popular alternative to MCMC methods. In theory, increasing the depth/complexity of the normalizing flow should lead to more accurate posterior approximations. However, in practice, training deep normalizing flows for high-dimensional posterior distributions is often infeasible due to the high variance of the stochastic gradients.

In the first part of this talk, I will explain various existing approaches for addressing high variance in stochastic gradients, and I will show that they can be insufficient to achieve convergence.

In the second part, I will introduce the log soft extension (LOFT) layer, which can effectively restrain the samples of NFs to lie in a reasonable range. I will show empirically that LOFT greatly reduces the variance of the ELBO (evidence lower bound) estimator, and leads to stable gradient estimates. For various different target distributions with high-dimensions or fat tails, we observe that LOFT enables successful training of NFs that was previously not possible.

Finally, I will show ongoing work when applying LOFT to approximating the posterior of a high-dimensional regression model with horseshoe prior.

Bio

Daniel Andrade is an Associate Professor at the Education and Research Center for Artificial Intelligence and Data Innovation at Hiroshima University. His research covers a broad field in applied machine learning, including natural language processing (NLP) and uncertainty quantification in various domains (e.g., medical and security). His current research interests are especially in text/numerical data analysis, model selection, Bayesian inference, and Bayesian statistics in general.

Andrade received his diploma in computer science from the University of Passau in Germany in 2007, his PhD in computer science from the University of Tokyo in 2011, and his PhD in statistics from the Graduate University for Advanced Studies (SOKENDAI) with the Institute of Statistical Mathematics in 2019. Before joining Hiroshima University in 2021, he worked as a researcher at NEC Central Research Laboratories, Japan.

Andrade has numerous publications in top machine learning conferences and journals. He is also a recipient of several awards, including the Special Industrial Achievement Award from the Information Processing Society of Japan, and a best paper award for his work on the Dempster-Shafer theory.