Blog

Scikit-Learn : Ridge Regression

  • Scikit-Learn ridge regression

Ridge regression or Tikhonov regularization is the regularization technique that performs L2 regularization. It modifies the loss function by adding the penalty (shrinkage quantity) equivalent to the square of the magnitude of coefficients.βˆ‘j=1m(Yiβˆ’W0βˆ’βˆ‘i=1nWiXji)2+Ξ±

  • sklearn.linear_model.Ridge is the module used to solve a regression model where loss function is the linear least squares function and regularization is L2.

Parameters

Following table consists the parameters used by Ridge module βˆ’

Sr.NoParameter & Description
1alpha βˆ’ {float, array-like}, shape(n_targets)Alpha is the tuning parameter that decides how much we want to penalize the model.
2fit_intercept βˆ’ BooleanThis parameter specifies that a constant (bias or intercept) should be added to the decision function. No intercept will be used in calculation, if it will set to false.
3tol βˆ’ float, optional, default=1e-4It represents the precision of the solution.
4normalize βˆ’ Boolean, optional, default = FalseIf this parameter is set to True, the regressor X will be normalized before regression. The normalization will be done by subtracting the mean and dividing it by L2 norm. If fit_intercept = False, this parameter will be ignored.
5copy_X βˆ’ Boolean, optional, default = TrueBy default, it is true which means X will be copied. But if it is set to false, X may be overwritten.
6max_iter βˆ’ int, optionalAs name suggest, it represents the maximum number of iterations taken for conjugate gradient solvers.
7solver βˆ’ str, {β€˜auto’, β€˜svd’, β€˜cholesky’, β€˜lsqr’, β€˜sparse_cg’, β€˜sag’, β€˜saga’}’This parameter represents which solver to use in the computational routines. Following are the properties of options under this parameterauto βˆ’ It let choose the solver automatically based on the type of data.svd βˆ’ In order to calculate the Ridge coefficients, this parameter uses a Singular Value Decomposition of X.cholesky βˆ’ This parameter uses the standard scipy.linalg.solve() function to get a closed-form solution.lsqr βˆ’ It is the fastest and uses the dedicated regularized least-squares routine scipy.sparse.linalg.lsqr.sag βˆ’ It uses iterative process and a Stochastic Average Gradient descent.saga βˆ’ It also uses iterative process and an improved Stochastic Average Gradient descent.
8random_state βˆ’ int, RandomState instance or None, optional, default = noneThis parameter represents the seed of the pseudo random number generated which is used while shuffling the data. Following are the options βˆ’int βˆ’ In this case, random_state is the seed used by random number generator.RandomState instance βˆ’ In this case, random_state is the random number generator.None βˆ’ In this case, the random number generator is the RandonState instance used by np.random.

Attributes

Followings table consist the attributes used by Ridge module βˆ’

Sr.NoAttributes & Description
1coef_ βˆ’ array, shape(n_features,) or (n_target, n_features)This attribute provides the weight vectors.
2Intercept_ βˆ’ float | array, shape = (n_targets)It represents the independent term in decision function.
3n_iter_ βˆ’ array or None, shape (n_targets)Available for only β€˜sag’ and β€˜lsqr’ solver, returns the actual number of iterations for each target.

Implementation Example

Following Python script provides a simple example of implementing Ridge Regression. We are using 15 samples and 10 features. The value of alpha is 0.5 in our case. There are two methods namely fit() and score() used to fit this model and calculate the score respectively.

from sklearn.linear_model import Ridge
import numpy as np
n_samples, n_features = 15, 10
rng = np.random.RandomState(0)
y = rng.randn(n_samples)
X = rng.randn(n_samples, n_features)
rdg = Ridge(alpha = 0.5)
rdg.fit(X, y)
rdg.score(X,y)

Output

0.76294987

The output shows that the above Ridge Regression model gave the score of around 76 percent. For more accuracy, we can increase the number of samples and features.

Example

For the above example, we can get the weight vector with the help of following python script βˆ’

rdg.coef_

Output

array([ 0.32720254, -0.34503436, -0.2913278 , 0.2693125 , -0.22832508,
   -0.8635094 , -0.17079403, -0.36288055, -0.17241081, -0.43136046])

Example

Similarly, we can get the value of intercept with the help of following python script βˆ’

rdg.intercept_

Output

0.527486

Next Topic : Click Here

This Post Has 2 Comments

Leave a Reply