Ridge Regression: R-Package (glmnet) vs Manual

The coefficients computed using glmnet are different from that of Manual computation.

This difference is primarily because of difference in objective function.

Manual Computation:

Objective Function: Theoretically objective function of ridge Regression:

Minimizing Sum square error by taking derivative and equating to zero:

Glmnet:

Objective Function:

 

 

i.          Y (Dependent variable) in glment objective function is implicitly divided by its standard deviation (s_y).

ii.          sy is computed by considering N the full sample size instead of n-1.

iii.           To recover estimates glmnet multiplies the estimates above by sy

iv.          Lambda the tuning parameter in glment is scaled by Sigmay / N:

λg = λm * SigmaY / N^2

v.          thresh = 1e-20: Defaults value of thresh is1e-7. Thresh argument is used for convergence threshold for coordinate descent. Each inner coordinate-descent loop continues until the maximum change in the objective after any coefficient update is less than thresh times the null deviance. 

vi.          Intercept is not penalized: The identity (diagonal) matrix will have first row and first column equivalent to 0.

vii.           Coefficients: glmnet returns the unstandardized coefficients i.e. coefficients to original scale. 

For glmnet (Standardize = T): Independent variables are standardized in the function. To unstandardized each coefficient is divided by its corresponding standard deviation using N.

Unstandardized Intercept = Standardized Intercept – Cross Product of Unstandardized coefficients computed above by their averages

Substituting (i) and (ii) in (5)

 

https://www.linkedin.com/feed/update/urn:li:activity:6555820683182604288