Blog

Here, we will learn about an optimization algorithm in Sklearn, termed as Stochastic Gradient Descent (SGD).

Stochastic Gradient Descent (SGD) is a simple yet efficient optimization algorithm used to find the values of parameters/coefficients of functions that minimize a cost function. In other words, it is used for discriminative learning of linear classifiers under convex loss functions such as SVM and Logistic regression. It has been successfully applied to large-scale datasets because the update to the coefficients is performed for each training instance, rather than at the end of instances.

SGD Classifier

Stochastic Gradient Descent (SGD) classifier basically implements a plain SGD learning routine supporting various loss functions and penalties for classification. Scikit-learn provides SGDClassifier module to implement SGD classification.

Parameters

Followings table consist the parameters used by SGDClassifier module −

Attributes

Following table consist the attributes used by SGDClassifier module −

Implementation Example

Like other classifiers, Stochastic Gradient Descent (SGD) has to be fitted with following two arrays −

• An array X holding the training samples. It is of size [n_samples, n_features].
• An array Y holding the target values i.e. class labels for the training samples. It is of size [n_samples].

Example

Following Python script uses SGDClassifier linear model −

import numpy as np
from sklearn import linear_model
X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1]])
Y = np.array([1, 1, 2, 2])
SGDClf = linear_model.SGDClassifier(max_iter = 1000, tol=1e-3,penalty = "elasticnet")
SGDClf.fit(X, Y)

Output

SGDClassifier(
   alpha = 0.0001, average = False, class_weight = None,
   early_stopping = False, epsilon = 0.1, eta0 = 0.0, fit_intercept = True,
   l1_ratio = 0.15, learning_rate = 'optimal', loss = 'hinge', max_iter = 1000,
   n_iter = None, n_iter_no_change = 5, n_jobs = None, penalty = 'elasticnet',
   power_t = 0.5, random_state = None, shuffle = True, tol = 0.001,
   validation_fraction = 0.1, verbose = 0, warm_start = False
)

Example

Now, once fitted, the model can predict new values as follows −

SGDClf.predict([[2.,2.]])

Output

array([2])

Example

For the above example, we can get the weight vector with the help of following python script −

SGDClf.coef_

Output

array([[19.54811198, 9.77200712]])

Example

Similarly, we can get the value of intercept with the help of following python script −

SGDClf.intercept_

Output

array([10.])

Example

We can get the signed distance to the hyperplane by using SGDClassifier.decision_function as used in the following python script −

SGDClf.decision_function([[2., 2.]])

Output

array([68.6402382])

SGD Regressor

Stochastic Gradient Descent (SGD) regressor basically implements a plain SGD learning routine supporting various loss functions and penalties to fit linear regression models. Scikit-learn provides SGDRegressor module to implement SGD regression.

Parameters

Parameters used by SGDRegressor are almost same as that were used in SGDClassifier module. The difference lies in ‘loss’ parameter. For SGDRegressor modules’ loss parameter the positives values are as follows −

• squared_loss − It refers to the ordinary least squares fit.
• huber: SGDRegressor − correct the outliers by switching from squared to linear loss past a distance of epsilon. The work of ‘huber’ is to modify ‘squared_loss’ so that algorithm focus less on correcting outliers.
• epsilon_insensitive − Actually, it ignores the errors less than epsilon.
• squared_epsilon_insensitive − It is same as epsilon_insensitive. The only difference is that it becomes squared loss past a tolerance of epsilon.

Another difference is that the parameter named ‘power_t’ has the default value of 0.25 rather than 0.5 as in SGDClassifier. Furthermore, it doesn’t have ‘class_weight’ and ‘n_jobs’ parameters.

Attributes

Attributes of SGDRegressor are also same as that were of SGDClassifier module. Rather it has three extra attributes as follows −

• average_coef_ − array, shape(n_features,)

As name suggest, it provides the average weights assigned to the features.

• average_intercept_ − array, shape(1,)

As name suggest, it provides the averaged intercept term.

• t_ − int

It provides the number of weight updates performed during the training phase.

Note − the attributes average_coef_ and average_intercept_ will work after enabling parameter ‘average’ to True.

Implementation Example

Following Python script uses SGDRegressor linear model −

import numpy as np
from sklearn import linear_model
n_samples, n_features = 10, 5
rng = np.random.RandomState(0)
y = rng.randn(n_samples)
X = rng.randn(n_samples, n_features)
SGDReg =linear_model.SGDRegressor(
   max_iter = 1000,penalty = "elasticnet",loss = 'huber',tol = 1e-3, average = True
)
SGDReg.fit(X, y)

Output

SGDRegressor(
   alpha = 0.0001, average = True, early_stopping = False, epsilon = 0.1,
   eta0 = 0.01, fit_intercept = True, l1_ratio = 0.15,
   learning_rate = 'invscaling', loss = 'huber', max_iter = 1000,
   n_iter = None, n_iter_no_change = 5, penalty = 'elasticnet', power_t = 0.25,
   random_state = None, shuffle = True, tol = 0.001, validation_fraction = 0.1,
   verbose = 0, warm_start = False
)

Example

Now, once fitted, we can get the weight vector with the help of following python script −

SGDReg.coef_

Output

array([-0.00423314, 0.00362922, -0.00380136, 0.00585455, 0.00396787])

Example

Similarly, we can get the value of intercept with the help of following python script −

SGReg.intercept_

Output

SGReg.intercept_

Example

We can get the number of weight updates during training phase with the help of the following python script −

SGDReg.t_

Output

61.0

Pros and Cons of SGD

Following the pros of SGD −

• Stochastic Gradient Descent (SGD) is very efficient.
• It is very easy to implement as there are lots of opportunities for code tuning.

Following the cons of SGD −

• Stochastic Gradient Descent (SGD) requires several hyperparameters like regularization parameters.
• It is sensitive to feature scaling.

Next Topic : Click Here