Linear Models

فهرست فصل‌ها دانلود آرشیوناشر

scikit-learn Docs · نسخهٔ 5e9184e53cd3 · BSD-3-Clause

=============

Linear Models

Linear models cover a wide range of different models, most of them regressors but also some important classifiers. What they all have in common is that their predictions are a linear combination of the p=n_{\text{features}} features, or a function thereof. This linear combination is learned from the data and parametrized by coefficients w. In mathematical notation, the predicted value \hat{y} can be written as

or, more generally, \hat{y}(w, x) = h(w_0 + w_1 x_1 + ...) with some inverse link or response function h. Across the module, we designate the vector w = (w_1, ..., w_p) as `coef_ and w_0 as intercept_`. For brevity, we denote the linear predictor as

such that it includes the intercept w_0. You might also think of X as the data matrix extended by a column of ones. In the implementation, scikit-learn tries to avoid as much as possible to build this extended matrix explicitly which would double the memory consumption. For norms of the coefficients as used in penalties, the intercept is always excluded, for example:

A consequence of the common linear predictor for all linear models is that each feature in X as passed to fit contributes a separate additive linear term. Non-linear terms and interactions of features must be added in the preprocessing as shown further down in Polynomial regression <polynomial_regression>. On top, it is considered good practice to standardize X before fitting a linear model.

To perform classification with generalized linear models, see Logistic_regression.

Ordinary Least Squares

LinearRegression predicts the conditional expectation E(y|X) by the linear predictor Xw. The coefficients and the intercept are fitted to minimize the residual sum of squares between the observed targets in the dataset, and the targets predicted by the linear approximation. Mathematically, it solves the least squares problem:

:target: ../auto_examples/linear_model/plot_ols_ridge.html :align: center :scale: 50%

LinearRegression takes in its `fit method arguments X, y, sample_weight and stores the coefficients w of the linear model in its coef_ and intercept_` attributes::

توجه: >> from sklearn.linear_model import LinearRegression >> reg = LinearRegression() >> reg.fit([[0, 0], [1, 1], [2, 2]], [0, 1, 2])

LinearRegression()

توجه: >> reg.coef_

array([0.5, 0.5])

توجه: >> reg.intercept_

0.0

When features are correlated, e.g. some columns of the design matrix X have an approximately linear dependence, the design matrix becomes ill-conditioned (close to singular) and as a result, the least squares estimate becomes highly sensitive to random errors or small changes in the observed target y, leading to a large inflation of the variance of the estimated coefficients. This situation of *multicollinearity* can arise, for example, when data are collected without an experimental design like in an observational study.

One possible remedy is to add a penalty for the size of the coefficients as in Ridge regression <ridge_regression>.

With sample weights s=\text{sample_weight}, LinearRegression minimizes the weighted squared error:

For dense X, the least squares solution is computed using the singular value decomposition of X via SciPy lstsq <scipy:scipy.linalg.lstsq>. For a matrix X of shape (n, p)=(n_samples, n_features) this method has a cost of O(\max(n, p)\min(n, p)^2).

For sparse X, the least squares problem is solved by the iterative solver SciPy lsqr <scipy:scipy.sparse.linalg.lsqr>.

  • sphx_glr_auto_examples_linear_model_plot_ols_ridge.py

Non-Negative Least Squares

It is possible to constrain all the coefficients to be non-negative, which may be useful when they represent some physical or naturally non-negative quantities (e.g., frequency counts or prices of goods). LinearRegression accepts a boolean `positive parameter: when set to True, Non-Negative Least Squares <https://en.wikipedia.org/wiki/Non-negative_least_squares>_ are then applied via SciPy nnls <scipy:scipy.optimize.nnls>`.

  • sphx_glr_auto_examples_linear_model_plot_nnls.py

Ridge regression and classification

Regression

Ridge regression addresses some of the problems of ordinary_least_squares by imposing a penalty on the size of the coefficients in the form of an \ell_2-norm. The ridge coefficients minimize a penalized residual sum of squares

\min_{w} || X w - y||_2^2 + \alpha ||w||_2^2

The penalty strength \alpha \geq 0 controls the amount of shrinkage of the estimated coefficients: the larger the value of \alpha, the greater the amount of shrinkage toward zero. This shrinkage causes a bias of the estimated coefficients but it offers at least two advantages w.r.t. ordinary least squares, see [HK1970]_:

  • The estimated coefficients become more robust to collinearity and ill-conditioning.
  • For an appropriate choice of alpha, the expected out-of-sample squared error

(*risk*) is lower, i.e. the expected prediction accuracy is better.

Because Ridge penalizes coefficient magnitudes, feature scales matter: for a comparable effect on the target, a feature with larger values can have a smaller coefficient and be penalized less. Standardizing <preprocessing_scaler> features makes the penalty more comparable across them.

If distances of cities of a country (in meters, typically several thousands) and travel times between cities (in hours, typically smaller 24) have similar effect on the target, Ridge will tend to shrink the travel time coefficient more when the features are unstandardized.

With sample weights s=\text{sample_weight}, Ridge minimizes:

:target: ../auto_examples/linear_model/plot_ridge_path.html :align: center :scale: 50%

As with other linear models, Ridge takes the arrays `X and y in its fit methods and stores the coefficients w in its attribute coef_ and w_0 as intercept_`::

توجه: >> from sklearn.linear_model import Ridge >> reg = Ridge(alpha=.5) >> reg.fit([[0, 0], [0, 0], [1, 1]], [0, .1, 1])

Ridge(alpha=0.5)

توجه: >> reg.coef_

array([0.34545455, 0.34545455])

توجه: >> reg.intercept_

np.float64(0.13636)

Note that the class Ridge implements different algorithms, all with different strengths and weaknesses, to solve the minimization problem. They are exposed as parameter solver. To ease this choice, the option solver="auto" automatically selects a suitable solver for the data and settings at hand. It will choose between the "lbfgs", "cholesky", and "sparse_cg" solvers. Ridge will begin checking the conditions shown in the following table from top to bottom. If the condition is true, the corresponding solver is chosen.

+-------------+----------------------------------------------------+

SolverCondition

+-------------+----------------------------------------------------+

'lbfgs'The `positive=True` option is specified.

+-------------+----------------------------------------------------+

'cholesky'The input array X is not sparse.

+-------------+----------------------------------------------------+

'sparse_cg'None of the above conditions are fulfilled.

+-------------+----------------------------------------------------+

The `"cholesky" and "sparse_cg" solvers use the primal formulation as given above when n_samples >= n_features. This means solving a system with n_features` unknowns:

(X^T X + \alpha I) w = X^T y.

When `n_samples < n_features, they use the dual formulation, solving a system with n_samples` unknowns and recovering the coefficients as follows:

(X X^T + \alpha I) c = y, \qquad w = X^T c.

Here, I is the identity matrix of the appropriate size, c denotes the dual coefficients, and X and y denote the data after mean centering and sample-weight rescaling, when applicable.

The `"cholesky" solver explicitly forms the corresponding Gram matrix X.T @ X. Storing the Gram matrix requires memory proportional to n_{\text{features}}^2 in the primal formulation and n_{\text{samples}}^2 in the dual formulation. In contrast, "sparse_cg"` uses matrix-vector products without forming the Gram matrix.

Ridge solvers have the same order of complexity as ordinary_least_squares. For a matrix X of shape (n, p)=(n_samples, n_features) the cost is O(\max(n, p)\min(n, p)^2). But depending on the selected solver, the constant factor, that is the effective numbers of flops, can be (significantly) smaller than the one of OLS.

  • sphx_glr_auto_examples_linear_model_plot_ols_ridge.py
  • sphx_glr_auto_examples_linear_model_plot_ridge_path.py
  • sphx_glr_auto_examples_inspection_plot_linear_model_coefficient_interpretation.py
  • sphx_glr_auto_examples_linear_model_plot_ridge_coeffs.py

Classification

The Ridge regressor has the classifier variant RidgeClassifier. This classifier first converts binary targets to `{-1, 1}` and then treats the problem as a regression task, optimizing the same objective as above. The predicted class corresponds to the sign of the regressor's prediction. For multiclass classification, the problem is treated as multi-output regression, and the predicted class corresponds to the output with the highest value.

It might seem questionable to use a (penalized) Least Squares loss to fit a classification model instead of the more traditional logistic or hinge losses. However, in practice, all those models can lead to similar cross-validation scores in terms of accuracy or precision/recall on data with a good (and linear) class separation. The advantage with the penalized least squares loss used by the RidgeClassifier is that it allows for a very different choice of the numerical solvers with distinct computational performance profiles.

The RidgeClassifier can be significantly faster than e.g. LogisticRegression with a high number of classes because it needs to compute the projection matrix (X^T X)^{-1} X^T only once.

This classifier is sometimes referred to as a Least Squares Support Vector Machine <https://en.wikipedia.org/wiki/Least-squares_support-vector_machine>_ with a linear kernel.

  • sphx_glr_auto_examples_text_plot_document_classification_20newsgroups.py

Setting the regularization parameter: leave-one-out Cross-Validation

RidgeCV and RidgeClassifierCV implement ridge regression/classification with built-in cross-validation of the alpha parameter. They work in the same way as ~sklearn.model_selection.GridSearchCV except that they default to efficient Leave-One-Out cross-validation. When using the default cross-validation, alpha cannot be 0 due to the formulation used to calculate Leave-One-Out error. See [RL2007]_ for details.

Usage example::

توجه: >> import numpy as np >> from sklearn.linear_model import RidgeCV >> reg = RidgeCV(alphas=np.logspace(-6, 6, 13)) >> reg.fit([[0, 0], [0, 0.1], [1, 1]], [0, -0.1, 1])

RidgeCV(alphas=array([1.e-06, 1.e-05, 1.e-04, 1.e-03, 1.e-02, 1.e-01, 1.e+00, 1.e+01, 1.e+02, 1.e+03, 1.e+04, 1.e+05, 1.e+06]))

توجه: >> reg.alpha_

np.float64(0.1)

Specifying the value of the cv attribute will trigger the use of cross-validation with ~sklearn.model_selection.GridSearchCV, for example cv=10 for 10-fold cross-validation, rather than Leave-One-Out Cross-Validation.

Estimation for Nonorthogonal Problems. Technometrics, 12(1), 55-67. <10.2307/1267351>`

<http://hdl.handle.net/1721.1/37318>_, course slides <https://www.mit.edu/~9.520/spring07/Classes/rlsslides.pdf>`_).

Lasso

The Lasso is a linear model that estimates sparse coefficients, meaning that it is able to set coefficients exactly to zero. A feature with a zero coefficient could be excluded from the model. Lasso is useful in some contexts due to its tendency to prefer solutions with fewer non-zero coefficients, effectively reducing the number of features upon which the given solution is dependent. For this reason, Lasso and its variants are fundamental to the field of compressed sensing. Under certain conditions, it can recover the exact set of non-zero coefficients (see sphx_glr_auto_examples_applications_plot_tomography_l1_reconstruction.py).

Mathematically, it consists of a linear model with an additional regularization term in the form of the \ell_1-norm of the coefficients. The objective function to minimize is:

The lasso estimate thus solves the least-squares with added penalty \alpha ||w||_1, where \alpha is a constant and ||w||_1 is the \ell_1-norm of the coefficient vector.

The objective function of the Lasso has a different normalization with `1/n_samples` in front of the squared error. The Ridge objective can be expressed in the same way and then reads

+ \frac{\alpha}{2n_{\text{samples}}} ||w||_2^2}

This shows that different estimators use a different scaling behaviour of the penalty parameter `alpha with the sample size n_samples. Thus the influence of alpha fades with increasing n_samples` for Ridge but not for Lasso.

With sample weights s=\text{sample_weight}, Lasso minimizes:

The implementation of the class Lasso uses coordinate descent <coordinate_descent> as the algorithm to fit the coefficients. See least_angle_regression for another implementation::

توجه: >> from sklearn.linear_model import Lasso >> reg = Lasso(alpha=0.1) >> reg.fit([[0, 0], [1, 1]], [0, 1])

Lasso(alpha=0.1)

توجه: >> reg.predict([[1, 1]])

array([0.8])

The function lasso_path is useful for lower-level tasks, as it computes the coefficients along the full path of possible values.

  • sphx_glr_auto_examples_linear_model_plot_lasso_and_elasticnet.py
  • sphx_glr_auto_examples_applications_plot_tomography_l1_reconstruction.py
  • sphx_glr_auto_examples_inspection_plot_linear_model_coefficient_interpretation.py
  • sphx_glr_auto_examples_linear_model_plot_lasso_model_selection.py

As the Lasso regression yields sparse models, it can thus be used to perform feature selection, as detailed in l1_feature_selection.

The following references explain the origin of the Lasso as well as properties of the Lasso problem and the duality gap computation used for convergence control.

  • :doi:`Robert Tibshirani. (1996) Regression Shrinkage and Selection Via the Lasso.

J. R. Stat. Soc. Ser. B Stat. Methodol., 58(1):267-288 <10.1111/j.2517-6161.1996.tb02080.x>`

  • "An Interior-Point Method for Large-Scale L1-Regularized Least Squares,"

S. J. Kim, K. Koh, M. Lustig, S. Boyd and D. Gorinevsky, in IEEE Journal of Selected Topics in Signal Processing, 2007 (Paper <https://web.stanford.edu/~boyd/papers/pdf/l1_ls.pdf>__)

Coordinate descent with gap safe screening rules

Coordinate descent (CD) is a strategy to solve a minimization problem that considers a single feature j at a time. This way, the optimization problem is reduced to a 1-dimensional problem which is easier to solve:

with index -j meaning all features but j. The solution is

with the soft-thresholding function S(z, \alpha) = \operatorname{sign}(z) \max(0, |z|-\alpha). Note that the soft-thresholding function is exactly zero whenever \alpha \geq |z|. The CD solver then loops over the features either in a cycle, picking one feature after the other in the order given by X (selection="cyclic"), or by randomly picking features (selection="random"). It stops if the duality gap is smaller than the provided tolerance tol.

To simplify the math, we assume here no intercept, and `y and X` are mean centered.

The duality gap G(w, v) is an upper bound of the difference between the current primal objective function of the Lasso, P(w), and its minimum P(w^\star), i.e. G(w, v) \geq P(w) - P(w^\star). It is given by G(w, v) = P(w) - D(v) with dual objective function

subject to ||X^Tv||_{\infty} \leq n_{\text{samples}}\alpha. At optimum, the duality gap is zero, G(w^\star, v^\star) = 0 (a property called strong duality). With (scaled) dual variable v = c r, current residual r = y - Xw and dual scaling

c = \begin{cases} 1, & ||X^Tr||_{\infty} \leq n_{\text{samples}}\alpha, \\ \frac{n_{\text{samples}}\alpha}{||X^Tr||_{\infty}}, & \text{otherwise} \end{cases}

the stopping criterion is

A clever method to speedup the coordinate descent algorithm is to screen features. The idea is to identify and exclude features with solution w_j = 0 as early as possible. Gap safe screening rules are such a tool. At any point during the optimization, they can tell if a feature can be safely excluded from further optimization and set to zero with certainty. The price for the certainty is that this rule does not identify all of them, but the closer to the optimum the more it identifies.

The first reference explains the coordinate descent solver used in scikit-learn, the others treat gap safe screening rules.

  • :doi:`Friedman, Hastie & Tibshirani. (2010).

Regularization Path For Generalized linear Models by Coordinate Descent. J Stat Softw 33(1), 1-22 <10.18637/jss.v033.i01>`

  • :arxiv:`O. Fercoq, A. Gramfort, J. Salmon. (2015).

Mind the duality gap: safer rules for the Lasso. Proceedings of Machine Learning Research 37:333-342, 2015. <1505.03410>`

  • :arxiv:`E. Ndiaye, O. Fercoq, A. Gramfort, J. Salmon. (2017).

Gap Safe Screening Rules for Sparsity Enforcing Penalties. Journal of Machine Learning Research 18(128):1-33, 2017. <1611.05780>`

Setting the regularization parameter

The `alpha parameter controls the degree of sparsity of the estimated coefficients. In contrast to Ridge, there is no computational shortcut for leave-one-out cross-validation. At least two good options remain to select alpha, both faster than a plain ~sklearn.model_selection.GridSearchCV: either compute cross-validation, which still benefits from fast alpha`-path computations, or use information based selection criteria.

Using cross-validation

Scikit-learn exposes estimators that select the Lasso `alpha parameter by cross-validation: LassoCV and LassoLarsCV. Both estimators exploit the fact that computing a full path of alpha` values over a single fold is relatively cheap.

For high-dimensional datasets with many collinear features, LassoCV with its coordinate descent solver is most often preferable. However, LassoLarsCV with its least angle algorithm <least_angle_regression> has the advantage of exploring more relevant values of alpha parameter, and if the number of samples is very small compared to the number of features, it is often faster than LassoCV.

:target: ../auto_examples/linear_model/plot_lasso_model_selection.html :scale: 48%

:target: ../auto_examples/linear_model/plot_lasso_model_selection.html :scale: 48%

Information-criteria based model selection

Alternatively, the estimator LassoLarsIC proposes to use the Akaike information criterion (AIC) or the Bayesian information criterion (BIC). It is a computationally cheaper alternative to find the optimal value of `alpha` as the regularization path is computed only once instead of k+1 times when using k-fold cross-validation.

Indeed, these criteria are computed on the in-sample training set. In short, they penalize the over-optimistic scores of the different Lasso models by their flexibility, see the *Mathematical details* of aic_bic.

However, such criteria need a proper estimation of the degrees of freedom of the solution, are derived for large samples (asymptotic results) and assume the correct model is among the candidates under investigation. They also tend to break when the problem is badly conditioned (e.g. more features than samples).

If the assumptions are met, AIC tends to select the model with optimal prediction performance, while BIC tends to identify the true sparse model if the true model is in the candidate list.

:target: ../auto_examples/linear_model/plot_lasso_lars_ic.html :align: center :scale: 50%

  • sphx_glr_auto_examples_linear_model_plot_lasso_model_selection.py
  • sphx_glr_auto_examples_linear_model_plot_lasso_lars_ic.py

AIC and BIC criteria

The definition of AIC, and thus BIC, may differ across the literature. In this section, we give more information regarding the criterion computed in scikit-learn.

The AIC criterion is defined as:

AIC = -2 \log(\mathcal{L}) + 2 \mathrm{df}

where \mathcal{L} is the maximum likelihood of the model and \mathrm{df} is the effective number of degrees of freedom. For the Lasso, \mathrm{df} is the number of non-zero coefficients (without penalty, `alpha=0, we would simply have df=n_features or n_features + 1` if the intercept is included).

The definition of BIC replaces the constant 2 by \log(n):

BIC = -2 \log(\mathcal{L}) + \log(n) \mathrm{df}

where n is the number of samples.

For a linear Gaussian model, the maximum log-likelihood is defined as:

\log(\mathcal{L}) = - \frac{n}{2} \log(2 \pi) - \frac{n}{2} \log(\sigma^2)

  • \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{2\sigma^2}

where \sigma^2 is the noise variance and is estimated from the given `y, y_i and \hat{y}_i are respectively the true and predicted targets, and n` is the number of samples.

Plugging the maximum log-likelihood in the AIC formula yields:

AIC = n \log(2 \pi \sigma^2) + \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sigma^2} + 2 \mathrm{df}

The first term of the above expression is sometimes discarded since it is a constant when \sigma^2 is provided. In addition, it is sometimes stated that the AIC is equivalent to the C_p statistic [12]_. In a strict sense, however, it is equivalent only up to some constant and a multiplicative factor.

At last, we mentioned above that \sigma^2 is an estimate of the noise variance. In LassoLarsIC when the parameter noise_variance is not provided (default), the noise variance is estimated via the unbiased estimator of the variance of the corresponding Ordinary Least Squares (OLS) model defined as:

\sigma^2 = \frac{\sum_{i=1}^{n} (y_i - \tilde{y}_i)^2}{n - p}

where \tilde{y}_i are the predictions of the OLS regression with p number of coefficients (also counting the intercept). Note that this formula is valid only when n > p.

"On the degrees of freedom of the lasso." The Annals of Statistics 35.5 (2007): 2173-2192. <0712.0881.pdf>`

Comparison with the regularization parameter of SVM

The equivalence between `alpha and the regularization parameter C of Support Vector Machines <svm> is given by alpha = 1 / C or alpha = 1 / (n_samples * C)`, depending on the estimator and the exact objective function optimized by the model.

Multi-task Lasso

The MultiTaskLasso is a linear model that estimates sparse coefficients for multiple regression problems, or tasks, jointly: `y is a 2-dimensional array of shape (n_samples, n_targets), in maths often expressed as capital letter Y. The coefficients are also 2-dimensional, W is of shape (n_features, n_targets) and accessible as the transpose of the coefficient attribute, coef_.T`. The constraint is that the selected features are the same for all the regression problems. This is enforced by the penalty, which couples each feature's coefficients across all tasks.

The following figure compares the location of the non-zero entries in the coefficient matrix W obtained with a simple Lasso or a MultiTaskLasso. The Lasso estimates yield scattered non-zeros while the non-zeros of the MultiTaskLasso are full columns.

:target: ../auto_examples/linear_model/plot_multi_task_lasso_support.html :scale: 48%

:target: ../auto_examples/linear_model/plot_multi_task_lasso_support.html :scale: 48%

  • sphx_glr_auto_examples_linear_model_plot_multi_task_lasso_support.py

Mathematically, it consists of a linear model trained with a mixed \ell_1 \ell_2-norm for regularization. The objective function to minimize is:

where \text{Fro} indicates the Frobenius norm

and \ell_1 \ell_2 reads

In both above norms, the index j is for the dimension of tasks. The implementation in the class MultiTaskLasso uses coordinate descent as the algorithm to fit the coefficients.

Elastic-Net

ElasticNet is a linear regression model that combines the \ell_2-norm penalty of Ridge with the \ell_1-norm penalty of Lasso. This combines the strengths of both: \ell_1 is able to produce sparse models with only a few non-zero coefficients while \ell_2 guards against numerical instabilities and can potentially achieve a better out-of-sample squared error loss. The convex combination of \ell_1 and \ell_2 is called elastic-net penalty. This mixing is controlled by the `l1_ratio` parameter.

Elastic-net is useful when there are multiple features that are correlated with one another. Lasso is likely to pick one of these at random, while elastic-net is likely to pick both.

With \rho=\text{l1_ratio}, the objective function to minimize is

\min_{w} { \frac{1}{2n_{\text{samples}}} ||X w - y||_2^2 + \alpha \rho ||w||_1 + \frac{\alpha(1-\rho)}{2} ||w||_2^2}

:target: ../auto_examples/linear_model/plot_lasso_lasso_lars_elasticnet_path.html :align: center :scale: 50%

The class ElasticNetCV can be used to set the parameters `alpha (\alpha) and l1_ratio (\rho`) by cross-validation.

  • sphx_glr_auto_examples_linear_model_plot_lasso_and_elasticnet.py
  • sphx_glr_auto_examples_linear_model_plot_lasso_lasso_lars_elasticnet_path.py
  • sphx_glr_auto_examples_linear_model_plot_elastic_net_precomputed_gram_matrix_with_weighted_samples.py

To simplify the math, we assume here no intercept, and `y and X` are mean centered.

Interestingly, there are two different dual formulations for Elastic-Net. The first one can be derived by starting with the Lasso and adding `n_features artificial observations to X and y to incorporate the \ell_2 penalty. The artificial observations come with n_features additional dual variables u`, and the dual objective function reads

  • \frac{\alpha(1-\rho)}{2} ||u||_2^2

with the dual feasible set :math:\{(v, u): ||X^Tv - n_{\text{samples}}\alpha (1-\rho)u||_{\infty} \leq n_{\text{samples}}\alpha \rho\}. At the optimum, v^\star = y - Xw^\star and u^\star = w^\star. As for the Lasso, a dual feasible point is obtained by rescaling the current residual and coefficients, (v, u) = c \cdot (y - Xw, w). The disadvantage is that for pure Ridge regression with \rho = \text{l1_ratio} = 0, this constraint becomes an equality and this simple rescaling does not yield a dual feasible point anymore.

The alternative formulation is derived by a direct computation of the dual and results in

  • \frac{1}{2n_{\text{samples}}^2 \alpha(1-\rho)}

\sum_j (\left|x_j \cdot v \right| - n_{\text{samples}}\alpha\rho)_+^2

Here, (x)_+= \max(x, 0) denotes the positive part and x_j the j-th column of `X. Surprisingly, the dual feasible set equals the whole of \mathbb{R}^{n_{\text{samples}}}, i.e. there are no restrictions on v. The disadvantage is that this formulation only holds for \alpha(1-\rho) > 0 (in particular l1_ratio must be smaller than 1). This second formulation is used in case of pure Ridge with l1_ratio=0` where the first dual version breaks down.

The first reference is the one that introduced the elastic-net, the second provides the alternative dual objective function. For references on the coordinate descent algorithm that underlies this estimator, see coordinate descent <coordinate_descent> above.

  • :doi:`Zou, H., & Hastie, T. (2005).

Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2), 301-320. <10.1111/j.1467-9868.2005.00503.x>`

  • :arxiv:`Dünner, C., Forte, S., Takáč, M., & Jaggi, M. (2016).

Primal-Dual Rates and Certificates. International Conference on Machine Learning. <1602.05205>`

Multi-task Elastic-Net

The MultiTaskElasticNet is the multi-task counterpart of ElasticNet: it fits multiple regression problems (tasks) jointly, with `Y being a 2D array of shape (n_samples, n_tasks). It extends MultiTaskLasso in the same way that ElasticNet extends Lasso, i.e. by adding a Ridge-like penalty. The mixed \ell_1 \ell_2-norm couples each feature's coefficients across all tasks such that the selected features are the same for all tasks. The additional squared Frobenius norm has the same beneficial properties as the \ell_2 penalty of ElasticNet`. As it is a plain sum of squares over all coefficients, it does not couple the tasks.

Mathematically, the multi-task elastic-net penalty is a convex combination of the \ell_1 \ell_2-norm and the squared Frobenius norm, controlled by \rho=\text{l1_ratio}. The objective function to minimize is:

\min_{W} { \frac{1}{2n_{\text{samples}}} ||X W - Y||_{\text{Fro}}^2 + \alpha \rho ||W||_{21} + \frac{\alpha(1-\rho)}{2} ||W||_{\text{Fro}}^2}

The implementation in the class MultiTaskElasticNet uses coordinate descent as the algorithm to fit the coefficients.

The class MultiTaskElasticNetCV can be used to set the parameters `alpha (\alpha) and l1_ratio (\rho`) by cross-validation.

Least Angle Regression

Least-angle regression (LARS) is a regression algorithm for high-dimensional data, developed by Bradley Efron, Trevor Hastie, Iain Johnstone and Robert Tibshirani. LARS is similar to forward stepwise regression. At each step, it finds the feature most correlated with the target. When there are multiple features having equal correlation, instead of continuing along the same feature, it proceeds in a direction equiangular between the features.

The advantages of LARS are:

  • It is numerically efficient in contexts where the number of features

is significantly greater than the number of samples.

  • It is computationally just as fast as forward selection and has

the same order of complexity as ordinary least squares.

  • It produces a full piecewise linear solution path, which is

useful in cross-validation or similar attempts to tune the model.

  • If two features are almost equally correlated with the target,

then their coefficients should increase at approximately the same rate. The algorithm thus behaves as intuition would expect, and also is more stable.

  • It is easily modified to produce solutions for other estimators,

like the Lasso.

The disadvantages of the LARS method include:

  • Because LARS is based upon an iterative refitting of the

residuals, it would appear to be especially sensitive to the effects of noise. This problem is discussed in detail by Weisberg in the discussion section of the Efron et al. (2004) Annals of Statistics article.

The LARS model can be used via the estimator Lars, or its low-level implementation lars_path or lars_path_gram.

LARS Lasso

LassoLars is a lasso model implemented using the LARS algorithm, and unlike the implementation based on coordinate descent, this yields the exact solution, which is piecewise linear as a function of the norm of its coefficients.

:target: ../auto_examples/linear_model/plot_lasso_lasso_lars_elasticnet_path.html :align: center :scale: 50%

::

توجه: >> from sklearn.linear_model import LassoLars >> reg = LassoLars(alpha=.1) >> reg.fit([[0, 0], [1, 1]], [0, 1])

LassoLars(alpha=0.1)

توجه: >> reg.coef_

array([0.6, 0. ])

  • sphx_glr_auto_examples_linear_model_plot_lasso_lasso_lars_elasticnet_path.py

The LARS algorithm provides the full path of the coefficients along the regularization parameter almost for free, thus a common operation is to retrieve the path with one of the functions lars_path or lars_path_gram.

The algorithm is similar to forward stepwise regression, but instead of including features at each step, the estimated coefficients are increased in a direction equiangular to each one's correlations with the residual.

Instead of giving a vector result, the LARS solution consists of a curve denoting the solution for each value of the \ell_1 norm of the parameter vector. The full coefficients path is stored in the array `coef_path_ of shape (n_features, max_features + 1)`. The first column is always zero.

  • Original Algorithm is detailed in the paper `Least Angle Regression

<https://hastie.su.domains/Papers/LARS/LeastAngle_2002.pdf>`_ by Hastie et al.

Orthogonal Matching Pursuit (OMP)

OrthogonalMatchingPursuit and orthogonal_mp implement the OMP algorithm for approximating the fit of a linear model with constraints imposed on the number of non-zero coefficients (i.e. the \ell_0 pseudo-norm).

Being a forward feature selection method like least_angle_regression, orthogonal matching pursuit can approximate the optimum solution vector with a fixed number of non-zero elements:

\underset{w}{\operatorname{arg\,min\,}} ||y - Xw||_2^2 \text{ subject to } ||w||_0 \leq n_{\text{nonzero_coefs}}

Alternatively, orthogonal matching pursuit can target a specific error instead of a specific number of non-zero coefficients. This can be expressed as:

\underset{w}{\operatorname{arg\,min\,}} ||w||_0 \text{ subject to } ||y-Xw||_2^2 \leq \text{tol}

OMP is based on a greedy algorithm that includes at each step the atom most highly correlated with the current residual. It is similar to the simpler matching pursuit (MP) method, but better in that at each iteration, the residual is recomputed using an orthogonal projection on the space of the previously chosen dictionary elements.

  • sphx_glr_auto_examples_linear_model_plot_omp.py
  • https://www.cs.technion.ac.il/~ronrubin/Publications/KSVD-OMP-v2.pdf
  • `Matching pursuits with time-frequency dictionaries

<https://www.di.ens.fr/~mallat/papiers/MallatPursuit93.pdf>`_, S. G. Mallat, Z. Zhang, 1993.

Bayesian Regression

Bayesian regression techniques can be used to include regularization parameters in the estimation procedure: the regularization parameter is not set in a hard sense but tuned to the data at hand.

This can be done by introducing uninformative priors <https://en.wikipedia.org/wiki/Non-informative_prior#Uninformative_priors>__ over the hyper parameters of the model. The \ell_{2} regularization used in ridge_regression is equivalent to finding a maximum a posteriori estimation under a Gaussian prior over the coefficients w with precision \lambda^{-1}. Instead of setting \lambda manually, it is possible to treat it as a random variable to be estimated from the data.

To obtain a fully probabilistic model, the output y is assumed to be Gaussian distributed around X w:

where \alpha is again treated as a random variable that is to be estimated from the data.

The advantages of Bayesian Regression are:

  • It adapts to the data at hand.
  • It can be used to include regularization parameters in the

estimation procedure.

The disadvantages of Bayesian regression include:

  • Inference of the model can be time consuming.
  • A good introduction to Bayesian methods is given in `C. Bishop: Pattern

Recognition and Machine Learning <https://www.microsoft.com/en-us/research/wp-content/uploads/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf>`__.

  • Original Algorithm is detailed in the book `Bayesian learning for neural

networks <https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=db869fa192a3222ae4f2d766674a378e47013b1b>`__ by Radford M. Neal.

Bayesian Ridge Regression

BayesianRidge estimates a probabilistic model of the regression problem as described above. The prior for the coefficient w is given by a spherical Gaussian:

\mathcal{N}(w|0,\lambda^{-1}\mathbf{I}_{p})

The priors over \alpha and \lambda are chosen to be gamma distributions <https://en.wikipedia.org/wiki/Gamma_distribution>__, the conjugate prior for the precision of the Gaussian. The resulting model is called *Bayesian Ridge Regression*, and is similar to the classical Ridge.

The parameters w, \alpha and \lambda are estimated jointly during the fit of the model, the regularization parameters \alpha and \lambda being estimated by maximizing the *log marginal likelihood*. The scikit-learn implementation is based on the algorithm described in Appendix A of (Tipping, 2001) where the update of the parameters \alpha and \lambda is done as suggested in (MacKay, 1992). The initial value of the maximization procedure can be set with the hyperparameters `alpha_init and lambda_init`.

There are four more hyperparameters, \alpha_1, \alpha_2, \lambda_1 and \lambda_2 of the gamma prior distributions over \alpha and \lambda. These are usually chosen to be *non-informative*. By default \alpha_1 = \alpha_2 = \lambda_1 = \lambda_2 = 10^{-6}.

Bayesian Ridge Regression is used for regression::

توجه: >> from sklearn.linear_model import BayesianRidge >> X = [[0., 0.], [1., 1.], [2., 2.], [3., 3.]] >> Y = [0., 1., 2., 3.] >> reg = BayesianRidge() >> reg.fit(X, Y)

BayesianRidge()

After being fitted, the model can then be used to predict new values::

توجه: >> reg.predict([[1, 0.]])

array([0.50000013])

The coefficients w of the model can be accessed::

توجه: >> reg.coef_

array([0.49999993, 0.49999993])

Due to the Bayesian framework, the weights found are slightly different from the ones found by ordinary_least_squares. However, Bayesian Ridge Regression is more robust to ill-posed problems.

  • sphx_glr_auto_examples_linear_model_plot_bayesian_ridge_curvefit.py
  • Section 3.3 in Christopher M. Bishop: Pattern Recognition and Machine Learning, 2006
  • David J. C. MacKay, Bayesian Interpolation <https://citeseerx.ist.psu.edu/doc_view/pid/b14c7cc3686e82ba40653c6dff178356a33e5e2c>_, 1992.
  • Michael E. Tipping, Sparse Bayesian Learning and the Relevance Vector Machine <https://www.jmlr.org/papers/volume1/tipping01a/tipping01a.pdf>_, 2001.

Automatic Relevance Determination - ARD

The Automatic Relevance Determination (as being implemented in ARDRegression) is a kind of linear model which is very similar to the Bayesian Ridge Regression_, but that leads to sparser coefficients w [1]_ [2]_.

ARDRegression poses a different prior over w: it drops the spherical Gaussian distribution for a centered elliptic Gaussian distribution. This means each coefficient w_{i} can itself be drawn from a Gaussian distribution, centered on zero and with a precision \lambda_{i}:

with A being a positive definite diagonal matrix and \text{diag}(A) = \lambda = \{\lambda_{1},...,\lambda_{p}\}.

In contrast to the Bayesian Ridge Regression_, each coordinate of w_{i} has its own standard deviation \frac{1}{\lambda_i}. The prior over all \lambda_i is chosen to be the same gamma distribution given by the hyperparameters \lambda_1 and \lambda_2.

ARD is also known in the literature as *Sparse Bayesian Learning* and *Relevance Vector Machine* [3]_ [4]_.

See sphx_glr_auto_examples_linear_model_plot_ard.py for a worked-out comparison between ARD and Bayesian Ridge Regression_.

See sphx_glr_auto_examples_linear_model_plot_lasso_and_elasticnet.py for a comparison between various methods - Lasso, ARD and ElasticNet - on correlated data.

Logistic regression

The logistic regression is implemented in LogisticRegression. Despite its name, it is implemented as a linear model for classification rather than regression in terms of the scikit-learn/ML nomenclature. The logistic regression is also known in the literature as logit regression, maximum-entropy classification (MaxEnt) or the log-linear classifier. In this model, the probabilities describing the possible outcomes of a single trial are modeled using a logistic function <https://en.wikipedia.org/wiki/Logistic_function>_.

This implementation can fit binary, One-vs-Rest, or multinomial logistic regression with optional \ell_1, \ell_2 or Elastic-Net regularization.

Regularization is applied by default, which is common in machine learning but not in statistics. Another advantage of regularization is that it improves numerical stability. No regularization amounts to setting alpha to zero.

Logistic regression is a special case of generalized_linear_models with a Binomial / Bernoulli conditional distribution and a Logit link. The numerical output of the logistic regression, which is the predicted probability, can be used as a classifier by applying a threshold (by default 0.5) to it. This is how it is implemented in scikit-learn, so it expects a categorical target, making the Logistic Regression a classifier.

  • sphx_glr_auto_examples_linear_model_plot_logistic_l1_l2_sparsity.py
  • sphx_glr_auto_examples_linear_model_plot_logistic_path.py
  • sphx_glr_auto_examples_linear_model_plot_logistic_multinomial.py
  • sphx_glr_auto_examples_linear_model_plot_sparse_logistic_regression_20newsgroups.py
  • sphx_glr_auto_examples_linear_model_plot_sparse_logistic_regression_mnist.py
  • sphx_glr_auto_examples_classification_plot_classification_probability.py

Binary Case

For notational ease, we assume that the target y_i takes values in the set \{0, 1\} for data point i. Once fitted, the ~sklearn.linear_model.LogisticRegression.predict_proba method of ~sklearn.linear_model.LogisticRegression predicts the probability of the positive class P(y_i=1|X_i) as

As an optimization problem, binary class logistic regression with regularization term r(w) minimizes the following objective function:

:name: regularized-logistic-loss

\min_{w} \frac{1}{S}\sum_{i=1}^n s_i \left(-y_i \log(\hat{p}(X_i)) - (1 - y_i) \log(1 - \hat{p}(X_i))\right) + \frac{\alpha}{S} r(w)\,,

where {s_i} corresponds to the weights assigned by the user to a specific training sample (the vector s is formed by element-wise multiplication of the class weights and sample weights), and the sum S = \sum_{i=1}^n s_i. For sample_weight=None (and no class weights), we have s_i=1 and S = n_{\text{samples}}.

We currently provide four choices for the regularization or penalty term r(w) via the arguments alpha and l1_ratio:

+-------------------------------+-------------------------------------------------+

penaltyr(w)

+===============================+=================================================+

none (alpha=0)0

+-------------------------------+-------------------------------------------------+

\ell_1 (l1_ratio=1)`\w\_1`

+-------------------------------+-------------------------------------------------+

\ell_2 (l1_ratio=0)`\frac{1}{2}\w\_2^2 = \frac{1}{2}w^T w`

+-------------------------------+-------------------------------------------------+

ElasticNet (0<l1_ratio<1)`\frac{1 - \rho}{2}w^T w + \rho \w\_1`

+-------------------------------+-------------------------------------------------+

For ElasticNet, \rho (which corresponds to the l1_ratio parameter) controls the strength of \ell_1 regularization vs. \ell_2 regularization. Elastic-Net is equivalent to \ell_1 when \rho = 1 and equivalent to \ell_2 when \rho=0.

Multinomial Case

The binary case can be extended to K classes leading to the multinomial logistic regression, see also log-linear model <https://en.wikipedia.org/wiki/Multinomial_logistic_regression#As_a_log-linear_model>_.

It is possible to parameterize a K-class classification model using only K-1 weight vectors, leaving one class probability fully determined by the other class probabilities by leveraging the fact that all class probabilities must sum to one. We deliberately choose to overparameterize the model using K weight vectors for ease of implementation and to preserve the symmetrical inductive bias regarding ordering of classes, see [16]_. This effect becomes especially important when using regularization. The choice of overparameterization can be detrimental for unpenalized models since then the solution may not be unique, as shown in [16]_.

Let y_i \in \{1, \ldots, K\} be the label (ordinal) encoded target variable for observation i. Instead of a single coefficient vector, we now have a matrix of coefficients W where each row vector W_k corresponds to class k. We aim at predicting the class probabilities P(y_i=k|X_i) via ~sklearn.linear_model.LogisticRegression.predict_proba as:

The objective for the optimization becomes

\min_W -\frac{1}{S}\sum_{i=1}^n \sum_{k=0}^{K-1} s_{ik} [y_i = k] \log(\hat{p}_k(X_i)) + \frac{\alpha}{S} r(W)\,,

where [P] represents the Iverson bracket which evaluates to 0 if P is false, otherwise it evaluates to 1.

Again, s_{ik} are the weights assigned by the user (multiplication of sample weights and class weights) with their sum S = \sum_{i=1}^n \sum_{k=0}^{K-1} s_{ik}. For sample_weight=None (and no class weights), we have s_i=1 and S = n_{\text{samples}}.

We currently provide four choices for the regularization or penalty term r(W) via the arguments alpha and l1_ratio, where m is the number of features:

+-------------------------------+----------------------------------------------------------------------------------+

penaltyr(W)

+===============================+==================================================================================+

none (alpha=0)0

+-------------------------------+----------------------------------------------------------------------------------+

\ell_1 (l1_ratio=1)`\W\_{1,1} = \sum_{i=1}^m\sum_{j=1}^{K}W_{i,j}`

+-------------------------------+----------------------------------------------------------------------------------+

\ell_2 (l1_ratio=0)`\frac{1}{2}\W\_F^2 = \frac{1}{2}\sum_{i=1}^m\sum_{j=1}^{K} W_{i,j}^2`

+-------------------------------+----------------------------------------------------------------------------------+

ElasticNet (0<l1_ratio<1)`\frac{1 - \rho}{2}\W\_F^2 + \rho \W\_{1,1}`

+-------------------------------+----------------------------------------------------------------------------------+

Solvers

The solvers implemented in the class LogisticRegression are "lbfgs", "liblinear", "newton-cholesky", "newton-cd", "newton-cd-gram", "newton-cg", "sag" and "saga":

The following table summarizes the penalties and multinomial multiclass supported by each solver:

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

Solvers

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

Penalties'lbfgs''liblinear''newton-cholesky''newton-cd''newton-cd-gram''newton-cg''sag''saga'

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

L2 penaltyyesyesyesyesyesyesyesyes

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

L1 penaltynoyesnoyesyesnonoyes

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

Elastic-Net (L1 + L2)nononoyesyesnonoyes

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

No penaltyyesnoyesyesyesyesyesyes

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

Multiclass support

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

multinomial multiclassyesnoyesyesyesyesyesyes

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

Behaviors

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

Penalize the intercept (bad)noyesnononononono

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

Faster for large datasetsnonononononoyesyes

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

Robust to unscaled datasetsyesyesyesyesyesyesnono

+------------------------------+-------------+-----------------+-----------------------+-----------------+----------------------+-----------------+-----------+------------+

The "lbfgs" solver is used by default for its robustness. For n_samples >> n_features, "newton-cholesky" is a good choice and can reach high precision (tiny tol values). With L1 or Elastic-Net penalty, "newton-cd-gram" is a good choice for n_samples >> n_features, while "newton-cd" is a good choice for n_features > n_samples (or both large). For large datasets the "saga" solver is usually faster (than "lbfgs"), in particular for low precision (high tol). For large dataset, you may also consider using SGDClassifier with loss="log_loss", which might be even faster but requires more tuning.

The following references explain some of the solvers.

  • For "lbfgs", see documentation of

SciPy minimize <scipy:scipy.optimize.minimize>

  • "liblinear"
  • Fan, R. E., Chang, K. W., Hsieh, C. J., Wang, X. R., & Lin, C. J. (2008).

LIBLINEAR: A library for large linear classification. the Journal of machine Learning research, 9, 1871-1874. https://www.jmlr.org/papers/volume9/fan08a/fan08a.pdf

  • :doi:`Yu, H. F., Huang, F. L., & Lin, C. J. (2011).

Dual coordinate descent methods for logistic regression and maximum entropy models. Machine Learning, 85(1), 41-75. <10.1007/s10994-010-5221-8>`

  • SAG -- :arxiv:`Schmidt, M., Le Roux, N., & Bach, F. (2017).

Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162(1), 83-112. <1309.2388>`

  • SAGA -- :arxiv:`Defazio, A., Bach, F., & Lacoste-Julien, S. (2014).

SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems, 27. <1407.0202>`

Differences between solvers

There might be a difference in the scores obtained between LogisticRegression with `solver=liblinear or ~sklearn.svm.LinearSVC and the external liblinear library directly, when fit_intercept=False and the fit coef_ (or) the data to be predicted are zeroes. This is because for the sample(s) with decision_function zero, LogisticRegression and ~sklearn.svm.LinearSVC predict the negative class, while liblinear predicts the positive class. Note that a model with fit_intercept=False and having many samples with decision_function zero, is likely to be an underfit, bad model and you are advised to set fit_intercept=True and increase the intercept_scaling`.

  • The solver "liblinear" uses a coordinate descent (CD) algorithm, and relies

on the excellent C++ LIBLINEAR library <https://www.csie.ntu.edu.tw/~cjlin/liblinear/>_, which is shipped with scikit-learn. However, the CD algorithm implemented in liblinear cannot learn a true multinomial (multiclass) model. If you still want to use "liblinear" on multiclass problems, you can use a "one-vs-rest" scheme OneVsRestClassifier(LogisticRegression(solver="liblinear")), see `~sklearn.multiclass.OneVsRestClassifier. Note that minimizing the multinomial loss is expected to give better calibrated results as compared to a "one-vs-rest" scheme. For \ell_1 regularization sklearn.svm.l1_min_c allows to calculate the lower bound for C, which equals 1/alpha in order to get a non "null" (all feature weights to zero) model.

  • The "lbfgs", "newton-cg", "newton-cholesky" and "sag" solvers only support

\ell_2 regularization or no regularization, and are found to converge faster for some high-dimensional data. These solvers (and "saga") learn a true multinomial logistic regression model [5]_.

  • The "sag" solver uses Stochastic Average Gradient descent [6]_. It is faster

than other solvers for large datasets, when both the number of samples and the number of features are large.

  • The "saga" solver [7]_ is a variant of "sag" that also supports the non-smooth

\ell_1 penalty (l1_ratio=1). This is therefore the solver of choice for sparse multinomial logistic regression. It is also the only solver that supports Elastic-Net (0 < l1_ratio < 1).

  • The "lbfgs" is an optimization algorithm that approximates the

Broyden–Fletcher–Goldfarb–Shanno algorithm [8]_, which belongs to quasi-Newton methods. As such, it can deal with a wide range of different training data and is therefore the default solver. Its performance, however, suffers on poorly scaled datasets and on datasets with one-hot encoded categorical features with rare categories.

  • The "newton-cholesky" solver is an exact Newton solver that calculates the Hessian

matrix and solves the resulting linear system. It is a very good choice for n_samples >> n_features and can reach high precision (tiny values of tol), but has a few shortcomings: Only \ell_2 regularization is supported. Furthermore, because the Hessian matrix is explicitly computed, the memory usage has a quadratic dependency on n_features as well as on n_classes.

For a comparison of some of these solvers, see [9]_.

:arxiv:SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives. <1407.0202>

<https://tminka.github.io/papers/logreg/minka-logreg.pdf>`_

"A Blockwise Descent Algorithm for Group-penalized Multiresponse and Multinomial Regression." <1311.6529>`

A logistic regression with \ell_1 penalty yields sparse models, and can thus be used to perform feature selection, as detailed in l1_feature_selection.

It is possible to obtain the p-values and confidence intervals for coefficients in cases of regression without penalization. The statsmodels package <https://pypi.org/project/statsmodels/>_ natively supports this. Within sklearn, one could use bootstrapping instead as well.

LogisticRegressionCV implements Logistic Regression with built-in cross-validation support, to find the optimal alpha and l1_ratio parameters according to the `scoring attribute. The "newton-cg", "sag", "saga" and "lbfgs" solvers are found to be faster for high-dimensional dense data, due to warm-starting (see Glossary <warm_start>`).

Generalized Linear Models

Generalized Linear Models (GLM) extend linear models in two ways [10]_. First, the predicted values \hat{y} are linked to a linear combination of the input variables X via an inverse link function h as

Secondly, the squared loss function is replaced by the unit deviance d of a distribution in the exponential family (or more precisely, a reproductive exponential dispersion model (EDM) [11]_).

The minimization problem becomes:

where \alpha is the L2 regularization penalty. When sample weights are provided, the average becomes a weighted average.

The following table lists some specific EDMs and their unit deviance :

================= ================================ ============================================ Distribution Target Domain Unit Deviance d(y, \hat{y}) ================= ================================ ============================================ Normal y \in (-\infty, \infty) (y-\hat{y})^2 Bernoulli y \in \{0, 1\} 2({y}\log\frac{y}{\hat{y}}+({1}-{y})\log\frac{{1}-{y}}{{1}-\hat{y}}) Categorical y \in \{0, 1, ..., k\} 2\sum_{i \in \{0, 1, ..., k\}} I(y = i) y_\text{i}\log\frac{I(y = i)}{\hat{I(y = i)}} Poisson y \in [0, \infty) 2(y\log\frac{y}{\hat{y}}-y+\hat{y}) Gamma y \in (0, \infty) 2(\log\frac{\hat{y}}{y}+\frac{y}{\hat{y}}-1) Inverse Gaussian y \in (0, \infty) \frac{(y-\hat{y})^2}{y\hat{y}^2} ================= ================================ ============================================

The Probability Density Functions (PDF) of these distributions are illustrated in the following figure,

:align: center :scale: 100%

PDF of a random variable Y following Poisson, Tweedie (power=1.5) and Gamma distributions with different mean values (\mu). Observe the point mass at Y=0 for the Poisson distribution and the Tweedie (power=1.5) distribution, but not for the Gamma distribution which has a strictly positive target domain.

The Bernoulli distribution is a discrete probability distribution modelling a Bernoulli trial - an event that has only two mutually exclusive outcomes. The Categorical distribution is a generalization of the Bernoulli distribution for a categorical random variable. While a random variable in a Bernoulli distribution has two possible outcomes, a Categorical random variable can take on one of K possible categories, with the probability of each category specified separately.

The choice of the distribution depends on the problem at hand:

  • If the target values y are counts (non-negative integer valued) or

relative frequencies (non-negative), you might use a Poisson distribution with a log-link.

  • If the target values are positive valued and skewed, you might try a Gamma

distribution with a log-link.

  • If the target values seem to be heavier tailed than a Gamma distribution, you

might try an Inverse Gaussian distribution (or even higher variance powers of the Tweedie family).

  • If the target values y are probabilities, you can use the Bernoulli

distribution. The Bernoulli distribution with a logit link can be used for binary classification. The Categorical distribution with a softmax link can be used for multiclass classification.

  • Agriculture / weather modeling: number of rain events per year (Poisson),

amount of rainfall per event (Gamma), total rainfall per year (Tweedie / Compound Poisson Gamma).

  • Risk modeling / insurance policy pricing: number of claim events /

policyholder per year (Poisson), cost per event (Gamma), total cost per policyholder per year (Tweedie / Compound Poisson Gamma).

  • Credit Default: probability that a loan can't be paid back (Bernoulli).
  • Fraud Detection: probability that a financial transaction like a cash transfer

is a fraudulent transaction (Bernoulli).

  • Predictive maintenance: number of production interruption events per year

(Poisson), duration of interruption (Gamma), total interruption time per year (Tweedie / Compound Poisson Gamma).

  • Medical Drug Testing: probability of curing a patient in a set of trials or

probability that a patient will experience side effects (Bernoulli).

  • News Classification: classification of news articles into three categories

namely Business News, Politics and Entertainment news (Categorical).

Second Edition. Boca Raton: Chapman and Hall/CRC. ISBN 0-412-31760-5.

and analysis of deviance. Monografias de matemática, no. 51. See also Exponential dispersion model. <https://en.wikipedia.org/wiki/Exponential_dispersion_model>_

Usage

TweedieRegressor implements a generalized linear model for the Tweedie distribution, that allows to model any of the above mentioned distributions using the appropriate `power` parameter. In particular:

  • `power = 0`: Normal distribution. Specific estimators such as

Ridge, ElasticNet are generally more appropriate in this case.

  • `power = 1: Poisson distribution. PoissonRegressor` is exposed

for convenience. However, it is strictly equivalent to TweedieRegressor(power=1, link='log').

  • `power = 2: Gamma distribution. GammaRegressor` is exposed for

convenience. However, it is strictly equivalent to TweedieRegressor(power=2, link='log').

  • `power = 3`: Inverse Gaussian distribution.

The link function is determined by the link parameter.

Usage example::

توجه: >> from sklearn.linear_model import TweedieRegressor >> reg = TweedieRegressor(power=1, alpha=0.5, link='log') >> reg.fit([[0, 0], [0, 1], [2, 2]], [0, 1, 2])

TweedieRegressor(alpha=0.5, link='log', power=1)

توجه: >> reg.coef_

array([0.2463, 0.4337])

توجه: >> reg.intercept_

np.float64(-0.7638)

  • sphx_glr_auto_examples_linear_model_plot_poisson_regression_non_normal_loss.py
  • sphx_glr_auto_examples_linear_model_plot_tweedie_regression_insurance_claims.py

The feature matrix X should be standardized before fitting. This ensures that the penalty treats features equally.

Since the linear predictor Xw can be negative and Poisson, Gamma and Inverse Gaussian distributions don't support negative values, it is necessary to apply an inverse link function that guarantees the non-negativeness. For example with link='log', the inverse link function becomes h(Xw)=\exp(Xw).

If you want to model a relative frequency, i.e. counts per exposure (time, volume, ...) you can do so by using a Poisson distribution and passing y=\frac{\mathrm{counts}}{\mathrm{exposure}} as target values together with \mathrm{exposure} as sample weights. For a concrete example see e.g. sphx_glr_auto_examples_linear_model_plot_tweedie_regression_insurance_claims.py.

When performing cross-validation for the power parameter of TweedieRegressor, it is advisable to specify an explicit scoring function, because the default scorer TweedieRegressor.score is a function of power itself.

Stochastic Gradient Descent - SGD

Stochastic gradient descent is a simple yet very efficient approach to fit linear models. It is particularly useful when the number of samples (and the number of features) is very large. The `partial_fit` method allows online/out-of-core learning.

The classes SGDClassifier and SGDRegressor provide functionality to fit linear models for classification and regression using different (convex) loss functions and different penalties. For instance, SGDClassifier with loss="log_loss" fits a logistic regression model, while with loss="hinge" it fits a linear support vector machine (SVM).

You can refer to the dedicated sgd documentation section for more details.

Perceptron

The Perceptron is another simple classification algorithm suitable for large scale learning and derives from SGD. By default:

  • It does not require a learning rate.
  • It is not regularized (penalized).
  • It updates its model only on mistakes.

The last characteristic implies that the Perceptron is slightly faster to train than SGD with the hinge loss and that the resulting models are sparser.

In fact, the Perceptron is a wrapper around the SGDClassifier class using a perceptron loss and a constant learning rate. Refer to mathematical section <sgd_mathematical_formulation> of the SGD procedure for more details.

Passive Aggressive Algorithms

The passive-aggressive (PA) algorithms are another family of 2 algorithms (PA-I and PA-II) for large-scale online learning that derive from SGD. They are similar to the Perceptron in that they do not require a learning rate. However, contrary to the Perceptron, they include a regularization parameter `eta0 (C` in the reference paper).

For classification, SGDClassifier(loss="hinge", penalty=None, learning_rate="pa1", eta0=1.0) can be used for PA-I or with `learning_rate="pa2" for PA-II. For regression, :class:SGDRegressor(loss="epsilon_insensitive", penalty=None, learning_rate="pa1", eta0=1.0) can be used for PA-I or with learning_rate="pa2"` for PA-II.

  • Crammer, K., Dekel, O., Keshet, J., Shalev-Shwartz, S. and Singer, Y. (2006).

"Online Passive-Aggressive Algorithms" <https://jmlr.org/papers/volume7/crammer06a/crammer06a.pdf>_ JMLR, 7(19), 551-585

Robustness regression: outliers and modeling errors

Robust regression aims to fit a regression model in the presence of corrupt data: either outliers, or error in the model.

:target: ../auto_examples/linear_model/plot_theilsen.html :scale: 50% :align: center

Different scenario and useful concepts

There are different things to keep in mind when dealing with data corrupted by outliers:

:target: ../auto_examples/linear_model/plot_robust_fit.html :scale: 60%

:target: ../auto_examples/linear_model/plot_robust_fit.html :scale: 60%

:target: ../auto_examples/linear_model/plot_robust_fit.html :scale: 60%

  • Outliers in X or in y?

==================================== ==================================== Outliers in the y direction Outliers in the X direction ==================================== ====================================

y_outliersX_outliers

==================================== ====================================

  • Fraction of outliers versus amplitude of error

The number of outlying points matters, but also how much they are outliers.

==================================== ==================================== Small outliers Large outliers ==================================== ====================================

y_outlierslarge_y_outliers

==================================== ====================================

An important notion of robust fitting is that of breakdown point: the fraction of data that can be outlying for the fit to start missing the inlying data.

Note that in general, robust fitting in high-dimensional setting (large n_features) is very hard. The robust models here will probably not work in these settings.

Scikit-learn provides 3 robust regression estimators: RANSAC <ransac_regression>, Theil Sen <theil_sen_regression> and HuberRegressor <huber_regression>.

  • HuberRegressor <huber_regression> should be faster than

RANSAC <ransac_regression> and Theil Sen <theil_sen_regression> unless the number of samples is very large, i.e. `n_samples >> n_features. This is because RANSAC <ransac_regression> and Theil Sen <theil_sen_regression> fit on smaller subsets of the data. However, both Theil Sen <theil_sen_regression> and RANSAC <ransac_regression> are unlikely to be as robust as HuberRegressor <huber_regression>` for the default parameters.

  • RANSAC <ransac_regression> is faster than Theil Sen <theil_sen_regression>

and scales much better with the number of samples.

  • RANSAC <ransac_regression> will deal better with large

outliers in the y direction (most common situation).

  • Theil Sen <theil_sen_regression> will cope better with

medium-size outliers in the X direction, but this property will disappear in high-dimensional settings.

When in doubt, use RANSAC <ransac_regression>.

RANSAC: RANdom SAmple Consensus

RANSAC (RANdom SAmple Consensus) fits a model from random subsets of inliers from the complete data set.

RANSAC is a non-deterministic algorithm producing only a reasonable result with a certain probability, which is dependent on the number of iterations (see max_trials parameter). It is typically used for linear and non-linear regression problems and is especially popular in the field of photogrammetric computer vision.

The algorithm splits the complete input sample data into a set of inliers, which may be subject to noise, and outliers, which are e.g. caused by erroneous measurements or invalid hypotheses about the data. The resulting model is then estimated only from the determined inliers.

:target: ../auto_examples/linear_model/plot_ransac.html :align: center :scale: 50%

  • sphx_glr_auto_examples_linear_model_plot_ransac.py
  • sphx_glr_auto_examples_linear_model_plot_robust_fit.py

Each iteration performs the following steps:

  1. Select `min_samples` random samples from the original data and check

whether the set of data is valid (see `is_data_valid`).

  1. Fit a model to the random subset (`estimator.fit`) and check

whether the estimated model is valid (see `is_model_valid`).

  1. Classify all data as inliers or outliers by calculating the residuals

to the estimated model (`estimator.predict(X) - y) - all data samples with absolute residuals smaller than or equal to the residual_threshold` are considered as inliers.

  1. Save fitted model as best model if number of inlier samples is

maximal. In case the current estimated model has the same number of inliers, it is only considered as the best model if it has better score.

These steps are performed either a maximum number of times (`max_trials) or until one of the special stop criteria are met (see stop_n_inliers and stop_score`). The final model is estimated using all inlier samples (consensus set) of the previously determined best model.

The `is_data_valid and is_model_valid functions allow to identify and reject degenerate combinations of random sub-samples. If the estimated model is not needed for identifying degenerate cases, is_data_valid` should be used as it is called prior to fitting the model and thus leading to better computational performance.

  • https://en.wikipedia.org/wiki/RANSAC
  • `"Random Sample Consensus: A Paradigm for Model Fitting with Applications to

Image Analysis and Automated Cartography" <https://www.cs.ait.ac.th/~mdailey/cvreadings/Fischler-RANSAC.pdf>`_ Martin A. Fischler and Robert C. Bolles - SRI International (1981)

  • `"Performance Evaluation of RANSAC Family"

<https://bmva-archive.org.uk/bmvc/2009/Papers/Paper355/Paper355.pdf>`_ Sunglok Choi, Taemin Kim and Wonpil Yu - BMVC (2009)

Theil-Sen estimator: generalized-median-based estimator

The TheilSenRegressor estimator uses a generalization of the median in multiple dimensions. It is thus robust to multivariate outliers. Note however that the robustness of the estimator decreases quickly with the dimensionality of the problem. It loses its robustness properties and becomes no better than an ordinary least squares in high dimension.

  • sphx_glr_auto_examples_linear_model_plot_theilsen.py
  • sphx_glr_auto_examples_linear_model_plot_robust_fit.py

TheilSenRegressor is comparable to the :ref:Ordinary Least Squares (OLS) <ordinary_least_squares> in terms of asymptotic efficiency and as an unbiased estimator. In contrast to OLS, Theil-Sen is a non-parametric method which means it makes no assumption about the underlying distribution of the data. Since Theil-Sen is a median-based estimator, it is more robust against corrupted data aka outliers. In univariate setting, Theil-Sen has a breakdown point of about 29.3% in case of a simple linear regression which means that it can tolerate arbitrary corrupted data of up to 29.3%.

:target: ../auto_examples/linear_model/plot_theilsen.html :align: center :scale: 50%

The implementation of TheilSenRegressor in scikit-learn follows a generalization to a multivariate linear regression model [#f1]_ using the spatial median which is a generalization of the median to multiple dimensions [#f2]_.

In terms of time and space complexity, Theil-Sen scales according to

\binom{n_{\text{samples}}}{n_{\text{subsamples}}}

which makes it infeasible to be applied exhaustively to problems with a large number of samples and features. Therefore, the magnitude of a subpopulation can be chosen to limit the time and space complexity by considering only a random subset of all possible combinations.

Also see the Wikipedia page <https://en.wikipedia.org/wiki/Theil%E2%80%93Sen_estimator>_

Huber Regression

The HuberRegressor is different from Ridge because it applies a linear loss to samples that are defined as outliers by the epsilon parameter. A sample is classified as an inlier if the absolute error of that sample is less than the threshold epsilon. It differs from TheilSenRegressor and RANSACRegressor because it does not ignore the effect of the outliers but gives a lesser weight to them.

:target: ../auto_examples/linear_model/plot_huber_vs_ridge.html :align: center :scale: 50%

  • sphx_glr_auto_examples_linear_model_plot_huber_vs_ridge.py

HuberRegressor minimizes

\min_{w, \sigma} {\sum_{i=1}^n\left(\sigma + H_{\epsilon}\left(\frac{X_{i}w - y_{i}}{\sigma}\right)\sigma\right) + \alpha {||w||_2}^2}

where the loss function is given by

H_{\epsilon}(z) = \begin{cases} z^2, & \text {if } |z| < \epsilon, \\ 2\epsilon|z| - \epsilon^2, & \text{otherwise} \end{cases}

It is advised to set the parameter `epsilon` to 1.35 to achieve 95% statistical efficiency.

  • Peter J. Huber, Elvezio M. Ronchetti: Robust Statistics, Concomitant scale

estimates, p. 172.

The HuberRegressor differs from using SGDRegressor with loss set to huber in the following ways.

  • HuberRegressor is scaling invariant. Once `epsilon is set, scaling X and y`

down or up by different values would produce the same robustness to outliers as before. as compared to SGDRegressor where `epsilon has to be set again when X and y` are scaled.

  • HuberRegressor should be more efficient to use on data with small number of

samples while SGDRegressor needs a number of passes on the training data to produce the same robustness.

Note that this estimator is different from the R implementation of Robust Regression <https://stats.oarc.ucla.edu/r/dae/robust-regression/>_ because the R implementation does a weighted least squares implementation with weights given to each sample on the basis of how much the residual is greater than a certain threshold.

Quantile Regression

Quantile regression estimates the median or other quantiles of y conditional on X, while ordinary least squares (OLS) estimates the conditional mean.

Quantile regression may be useful if one is interested in predicting an interval instead of point prediction. Sometimes, prediction intervals are calculated based on the assumption that prediction error is distributed normally with zero mean and constant variance. Quantile regression provides sensible prediction intervals even for errors with non-constant (but predictable) variance or non-normal distribution.

:target: ../auto_examples/linear_model/plot_quantile_regression.html :align: center :scale: 50%

Based on minimizing the pinball loss, conditional quantiles can also be estimated by models other than linear models. For example, ~sklearn.ensemble.GradientBoostingRegressor can predict conditional quantiles if its parameter `loss is set to "quantile" and parameter alpha is set to the quantile that should be predicted. See the example in sphx_glr_auto_examples_ensemble_plot_gradient_boosting_quantile.py`.

Most implementations of quantile regression are based on linear programming problem. The current implementation is based on scipy.optimize.linprog.

  • sphx_glr_auto_examples_linear_model_plot_quantile_regression.py

As a linear model, the QuantileRegressor gives linear predictions \hat{y}(w, X) = Xw for the q-th quantile, q \in (0, 1). The weights or coefficients w are then found by the following minimization problem:

\min_{w} {\frac{1}{n_{\text{samples}}} \sum_i PB_q(y_i - X_i w) + \alpha ||w||_1}.

This consists of the pinball loss (also known as linear loss), see also ~sklearn.metrics.mean_pinball_loss,

PB_q(t) = q \max(t, 0) + (1 - q) \max(-t, 0) = \begin{cases} q t, & t > 0, \\ 0, & t = 0, \\ (q-1) t, & t < 0 \end{cases}

and the L1 penalty controlled by parameter `alpha, similar to Lasso`.

As the pinball loss is only linear in the residuals, quantile regression is much more robust to outliers than squared error based estimation of the mean. Somewhat in between is the HuberRegressor.

  • Koenker, R., & Bassett Jr, G. (1978). `Regression quantiles.

<https://gib.people.uic.edu/RQ.pdf>`_ Econometrica: journal of the Econometric Society, 33-50.

  • Portnoy, S., & Koenker, R. (1997). :doi:`The Gaussian hare and the Laplacian

tortoise: computability of squared-error versus absolute-error estimators. Statistical Science, 12, 279-300 <10.1214/ss/1030037960>`.

  • Koenker, R. (2005). Quantile Regression <10.1017/CBO9780511754098>.

Cambridge University Press.

Polynomial regression: extending linear models with basis functions

One common pattern within machine learning is to use linear models trained on nonlinear functions of the data. This approach maintains the generally fast performance of linear methods, while allowing them to fit a much wider range of data.

For example, a simple linear regression can be extended by constructing polynomial features from the coefficients. In the standard linear regression case, you might have a model that looks like this for two-dimensional data:

If we want to fit a paraboloid to the data instead of a plane, we can combine the features in second-order polynomials, so that the model looks like this:

The (sometimes surprising) observation is that this is *still a linear model*: to see this, imagine creating a new set of features

With this re-labeling of the data, our problem can be written

We see that the resulting *polynomial regression* is in the same class of linear models we considered above (i.e. the model is linear in w) and can be solved by the same techniques. By considering linear fits within a higher-dimensional space built with these basis functions, the model has the flexibility to fit a much broader range of data.

Here is an example of applying this idea to one-dimensional data, using polynomial features of varying degrees:

:target: ../auto_examples/linear_model/plot_polynomial_interpolation.html :align: center :scale: 50%

This figure is created using the PolynomialFeatures transformer, which transforms an input data matrix into a new data matrix of a given degree. It can be used as follows::

توجه: >> from sklearn.preprocessing import PolynomialFeatures >> import numpy as np >> X = np.arange(6).reshape(3, 2) >> X

array([[0, 1], [2, 3], [4, 5]])

توجه: >> poly = PolynomialFeatures(degree=2) >> poly.fit_transform(X)

array([[ 1., 0., 1., 0., 0., 1.], [ 1., 2., 3., 4., 6., 9.], [ 1., 4., 5., 16., 20., 25.]])

The features of `X have been transformed from [x_1, x_2] to [1, x_1, x_2, x_1^2, x_1 x_2, x_2^2]`, and can now be used within any linear model.

This sort of preprocessing can be streamlined with the Pipeline <pipeline> tools. A single object representing a simple polynomial regression can be created and used as follows::

توجه: >> from sklearn.preprocessing import PolynomialFeatures >> from sklearn.linear_model import LinearRegression >> from sklearn.pipeline import Pipeline >> import numpy as np >> model = Pipeline([('poly', PolynomialFeatures(degree=3)),

... ('linear', LinearRegression(fit_intercept=False))])

توجه: >> # fit to an order-3 polynomial data >> x = np.arange(5) >> y = 3 - 2 * x + x 2 - x 3 >> model = model.fit(x[:, np.newaxis], y) >> model.named_steps['linear'].coef_

array([ 3., -2., 1., -1.])

The linear model trained on polynomial features is able to exactly recover the input polynomial coefficients.

In some cases it's not necessary to include higher powers of any single feature, but only the so-called *interaction features* that multiply together at most d distinct features. These can be gotten from PolynomialFeatures with the setting `interaction_only=True`.

For example, when dealing with boolean features, x_i^n = x_i for all n and is therefore useless; but x_i x_j represents the conjunction of two booleans. This way, we can solve the XOR problem with a linear classifier::

توجه: >> from sklearn.linear_model import Perceptron >> from sklearn.preprocessing import PolynomialFeatures >> import numpy as np >> X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]) >> y = X[:, 0] ^ X[:, 1] >> y

array([0, 1, 1, 0])

توجه: >> X = PolynomialFeatures(interaction_only=True).fit_transform(X).astype(int) >> X

array([[1, 0, 0, 0], [1, 0, 1, 0], [1, 1, 0, 0], [1, 1, 1, 1]])

توجه: >> clf = Perceptron(fit_intercept=False, max_iter=10, tol=None,

... shuffle=False).fit(X, y)

And the classifier "predictions" are perfect::

توجه: >> clf.predict(X)

array([0, 1, 1, 0])

توجه: >> clf.score(X, y)

1.0