Sarah Whelchel

Western Washington University
, Online (Zoom)

Abstract

Model Selection and Estimation via Ridge and Lasso Regression

The ordinary least squares (OLS) approach to finding parameter estimates in a linear regression requires the design matrix to have full rank. In practice, it is not uncommon to model with large datasets where the number of predictors is far greater than the number of observations, i.e., the design matrix is not full rank. Although OLS cannot be used in situations where our design matrix does not have full rank, depending on the constraint region added on the parameter space, unique model estimation, and possibly model selection, are achievable. Here, the lasso and ridge regression aim to minimize the residual sum of squares (RSS) subject to the absolute and square norm constraint on the parameter space, respectively. Interestingly, the ridge regression always estimates parameters uniquely, regardless of the design matrix given. On the other hand, the lasso regression allows for model selection on top of parameter estimation, which is crucial for producing interpretable models when dealing with large datasets. In this presentation, derivation of the lasso and ridge regression estimators, comparison of their variances to that of OLS, and a real-life example will be discussed.