Empirical Bayes Estimation for L1 Regularization: A Detailed Analysis in the One-Parameter Lasso Model

Tsukasa YOSHIDA  Kazuho WATANABE  

Publication
IEICE TRANSACTIONS on Fundamentals of Electronics, Communications and Computer Sciences   Vol.E101-A   No.12   pp.2184-2191
Publication Date: 2018/12/01
Online ISSN: 1745-1337
DOI: 10.1587/transfun.E101.A.2184
Type of Manuscript: Special Section PAPER (Special Section on Information Theory and Its Applications)
Category: Machine learning
Keyword: 
lasso regression,  empirical Bayes,  Laplace prior,  local variational approximation,  

Full Text: PDF(880.1KB)
>>Buy this Article


Summary: 
Lasso regression based on the L1 regularization is one of the most popular sparse estimation methods. It is often required to set appropriately in advance the regularization parameter that determines the degree of regularization. Although the empirical Bayes approach provides an effective method to estimate the regularization parameter, its solution has yet to be fully investigated in the lasso regression model. In this study, we analyze the empirical Bayes estimator of the one-parameter model of lasso regression and show its uniqueness and its properties. Furthermore, we compare this estimator with that of the variational approximation, and its accuracy is evaluated.