New Methods for Regularization Path Optimization via Differential Equations

Heyuan Liu, Paul Grigas. “New Methods for Regularization Path Optimization via Differential Equations“. NeurIPS 2019 Workshop on Beyond First Order Methods in Machine Learning. 

We develop and analyze several second order algorithms for computing an approximately optimal solution (regularization) path of a parameterized convex optimization problem with smooth Hessian. Our algorithms are inspired by a differential equations perspective on the parametric solution path and do not rely on the specific structure of the regularizer. We present computational guarantees that bound the oracle complexity to achieve an approximately optimal solution path under different smoothness assumptions and that also hold in the presence of approximate subproblems. We conduct numerical experiments that demonstrate the
viability of our approach, especially in the presence of higher-order smoothness.