Comparison among Ordinary Least Squares, Ridge, Lasso, and Elastic Net Estimators in the Presence of Outliers: Simulation and Application

Abstract

In linear regression models, multicollinearity often results in unstable and unreliable parameter estimates. Ridge regression, a biased estimation technique, is commonly used to mitigate this issue and produce more reliable estimates of regression coefficients. Several estimators have been developed to select the optimal ridge parameter. This study focuses on the top 16 estimators from the 366 evaluated by Mermi et al. (2024), along with seven additional estimators introduced over time. These 23 estimators were compared to Ordinary Least Squares (OLS), Elastic-Net (EN), Lasso, and generalized ridge (GR) regression, to evaluate their performance across different levels of multicollinearity in multiple regression settings. Simulated data, both with and without outliers, and various parametric conditions were used for the comparisons. The results indicated that certain ridge regression estimators perform reliably with small sample sizes and high correlations (around 0.95) in the absence of outliers. However, when outliers were present, some estimators performed better due to small sample sizes and increased variance. Furthermore, GR, EN, and Lasso exhibited robustness with large datasets, except in cases with substantial outliers and high variance.

IJSS, Vol. 24(2) (Special Issue), December, 2024, pp 25-48