Imputation of missing values

فهرست فصل‌ها دانلود آرشیوناشر

scikit-learn Docs · نسخهٔ 5e9184e53cd3 · BSD-3-Clause

============================

Imputation of missing values

  • Avoid dropping rows with missing values; it risks creating bias.
  • Some supervised learning methods <estimators_that_handle_nan> (typically

tree-based learners) can natively predict on data with missing values and work fairly well with no additional cost.

  • Imputation can be very computationaly costly, and quickly hits

diminishing returns to improve subsequent prediction performance [3]_.

For various reasons, many real-world datasets contain missing values, often encoded as blanks, NaNs or other placeholders. They arise from faulty measurements, unanswered questionnaire items, or information that was simply never recorded, and can affect both the data `X and the target y`. Most scikit-learn estimators assume that every entry of an array is numerical and holds meaning, and therefore cannot be trained directly on incomplete data.

A naive way to meet that requirement is to discard every row or column that contains a missing value. This is detrimental on two counts: valuable information from those cases is lost, and it generally introduces bias, since the remaining samples are rarely representative of the original population (unless values are missing completely at random). The same caution applies to the target: silently dropping samples whose outcome `y is unknown biases the analysis. In particular, when an outcome has not been observed yet, it is a censoring problem <https://en.wikipedia.org/wiki/Survival_analysis#Censoring>_ and should be handled with dedicated methods from survival analysis <https://en.wikipedia.org/wiki/Survival_analysis>`_ rather than by discarding data.

The tools described on this page address missing values in `X. A better strategy than discarding data is to impute the missing values, i.e. to infer them from the known part of the data (see the glossary entry on imputation). scikit-learn provides simple column statistics with SimpleImputer, as well as the model-based IterativeImputer and KNNImputer`.

Invest in imputation quality mainly when reconstructing the data itself, not prediction, is the objective. When the goal is prediction rather than reconstructing the data, a few high-level guidelines help choose among these tools [3]_:

  • Start simple: constant or mean / most-frequent imputation with

SimpleImputer is a strong and cheap baseline, and more elaborate imputation often brings only marginal gains in predictive performance.

  • Flag missing entries: adding a missingness indicator (the `add_indicator`

option of the imputers, or MissingIndicator) tends to help prediction, even when values are missing completely at random.

  • Prefer expressive models for the supervised-learner step: flexible estimators

benefit even less from sophisticated imputation, and some handle missing values natively without any imputation (see estimators_that_handle_nan).

  • Sophisticated imputation can help prediction, but it can be very

computationally cost (it scales poorly with data size).

  • Invest heavily in imputation quality mainly when reconstructing the data

itself, not prediction, is the objective.

Univariate vs. Multivariate Imputation

One type of imputation algorithm is univariate, which imputes values in the i-th feature dimension using only non-missing values in that feature dimension (e.g. SimpleImputer). By contrast, multivariate imputation algorithms use the entire set of available feature dimensions to estimate the missing values (e.g. IterativeImputer).

Univariate feature imputation

The SimpleImputer class provides basic strategies for imputing missing values. Missing values can be imputed with a provided constant value, or using the statistics (mean, median or most frequent) of each column in which the missing values are located. This class also allows for different missing values encodings.

The following snippet demonstrates how to replace missing values, encoded as `np.nan`, using the mean value of the columns (axis 0) that contain the missing values::

توجه: >> import numpy as np >> from sklearn.impute import SimpleImputer >> imp = SimpleImputer(missing_values=np.nan, strategy='mean') >> imp.fit([[1, 2], [np.nan, 3], [7, 6]])

SimpleImputer()

توجه: >> X = [[np.nan, 2], [6, np.nan], [7, 6]] >> print(imp.transform(X))

[[4. 2. ] [6. 3.666] [7. 6. ]]

The SimpleImputer class also supports sparse matrices::

توجه: >> import scipy.sparse as sp >> X = sp.csc_array([[1, 2], [0, -1], [8, 4]]) >> imp = SimpleImputer(missing_values=-1, strategy='mean') >> imp.fit(X)

SimpleImputer(missing_values=-1)

توجه: >> X_test = sp.csc_array([[-1, 2], [6, -1], [7, 6]]) >> print(imp.transform(X_test).toarray())

[[3. 2.] [6. 3.] [7. 6.]]

Note that this format is not meant to be used to implicitly store missing values in the matrix because it would densify it at transform time. Missing values encoded by 0 must be used with dense input.

The SimpleImputer class also supports categorical data represented as string values or pandas categoricals when using the `'most_frequent' or 'constant'` strategy::

توجه: >> import pandas as pd >> df = pd.DataFrame([["a", "x"],

... [np.nan, "y"], ... ["a", np.nan], ... ["b", "y"]], dtype="category") ...

توجه: >> imp = SimpleImputer(strategy="most_frequent") >> print(imp.fit_transform(df))

[['a' 'x'] ['a' 'y'] ['a' 'y'] ['b' 'y']]

For another example on usage, see sphx_glr_auto_examples_impute_plot_missing_values.py.

Multivariate feature imputation

A more sophisticated approach is to use the IterativeImputer class, which models each feature with missing values as a function of other features, and uses that estimate for imputation. It does so in an iterated round-robin fashion: at each step, a feature column is designated as output `y and the other feature columns are treated as inputs X. A regressor is fit on (X, y) for known y. Then, the regressor is used to predict the missing values of y. This is done for each feature in an iterative fashion, and then is repeated for max_iter` imputation rounds. The results of the final imputation round are returned.

::

توجه: >> import numpy as np >> from sklearn.impute import IterativeImputer >> imp = IterativeImputer(max_iter=10, random_state=0) >> imp.fit([[1, 2], [3, 6], [4, 8], [np.nan, 3], [7, np.nan]])

IterativeImputer(random_state=0)

توجه: >> X_test = [[np.nan, 2], [6, np.nan], [np.nan, 6]] >> # the model learns that the second feature is double the first >> print(np.round(imp.transform(X_test)))

[[ 1. 2.] [ 6. 12.] [ 3. 6.]]

Both SimpleImputer and IterativeImputer can be used in a Pipeline as a way to build a composite estimator that supports imputation. See sphx_glr_auto_examples_impute_plot_missing_values.py.

Convergence and diminishing returns

IterativeImputer repeats the round-robin imputation for `max_iter rounds and stops early when the change between two consecutive rounds falls below tol (when sample_posterior=False`). In practice, the imputed values often do not converge: the round-robin scheme is not guaranteed to reach a fixed point, so the number of iterations is best seen as a trade-off against the available time budget rather than a target to be met.

This is rarely a problem when the goal is prediction. Le Morvan and Varoquaux [3]_ report strongly diminishing returns: better imputation accuracy yields only small gains in downstream predictive performance, especially with an expressive model and a missingness indicator (`add_indicator`).

In practice, prefer a small, fixed `max_iter (i.e. max_iter=10) together with a missingness indicator and an expressive downstream model over investing in convergence. When imputation itself is the goal (e.i. for *reconstructing data*) rather than predicting, the number of iterations (i.e. max_iter>50`) and the choice of imputer matter more and should be assessed on the task at hand.

Flexibility of IterativeImputer

There are many well-established imputation packages in the R data science ecosystem: Amelia, mi, mice, missForest, etc. missForest is popular, and turns out to be a particular instance of different sequential imputation algorithms that can all be implemented with IterativeImputer by passing in different regressors to be used for predicting missing feature values. In the case of missForest, this regressor is a Random Forest. See sphx_glr_auto_examples_impute_plot_iterative_imputer_variants_comparison.py.

Multiple vs. Single Imputation

In the statistics community, it is common practice to perform multiple imputations, generating, for example, `m separate imputations for a single feature matrix. Each of these m imputations is then put through the subsequent analysis pipeline (e.g. feature engineering, clustering, regression, classification). The m` final analysis results (e.g. held-out validation errors) allow the data scientist to obtain understanding of how analytic results may differ as a consequence of the inherent uncertainty caused by the missing values. The above practice is called multiple imputation.

Our implementation of IterativeImputer was inspired by the R MICE package (Multivariate Imputation by Chained Equations) [1]_, but differs from it by returning a single imputation instead of multiple imputations. However, IterativeImputer can also be used for multiple imputations by applying it repeatedly to the same dataset with different random seeds when `sample_posterior=True`. See [2]_, chapter 4 for more discussion on multiple vs. single imputations.

It is still an open problem as to how useful single vs. multiple imputation is in the context of prediction and classification when the user is not interested in measuring uncertainty due to missing values.

Note that a call to the `transform method of IterativeImputer is not allowed to change the number of samples. Therefore multiple imputations cannot be achieved by a single call to transform`.

Imputation by Chained Equations in R". Journal of Statistical Software 45: 1-67. <https://www.jstatsoft.org/article/view/v045i03>`_

with Missing Data". John Wiley & Sons, Inc., New York, NY, USA.

beware of diminishing returns". International Conference on Learning Representations (ICLR). <https://arxiv.org/abs/2407.19804>`_

Nearest neighbors imputation

The KNNImputer class provides imputation for filling in missing values using the k-Nearest Neighbors approach. By default, a euclidean distance metric that supports missing values, ~sklearn.metrics.pairwise.nan_euclidean_distances, is used to find the nearest neighbors. Each missing feature is imputed using values from `n_neighbors nearest neighbors that have a value for the feature. The feature of the neighbors are averaged uniformly or weighted by distance to each neighbor. If a sample has more than one feature missing, then the neighbors for that sample can be different depending on the particular feature being imputed. When the number of available neighbors is less than n_neighbors and there are no defined distances to the training set, the training set average for that feature is used during imputation. If there is at least one neighbor with a defined distance, the weighted or unweighted average of the remaining neighbors will be used during imputation. If a feature is always missing in training, it is removed during transform`. For more information on the methodology, see ref. [OL2001]_.

The following snippet demonstrates how to replace missing values, encoded as `np.nan`, using the mean feature value of the two nearest neighbors of samples with missing values::

توجه: >> import numpy as np >> from sklearn.impute import KNNImputer >> nan = np.nan >> X = [[1, 2, nan], [3, 4, 3], [nan, 6, 5], [8, 8, 7]] >> imputer = KNNImputer(n_neighbors=2, weights="uniform") >> imputer.fit_transform(X)

array([[1. , 2. , 4. ], [3. , 4. , 3. ], [5.5, 6. , 5. ], [8. , 8. , 7. ]])

For another example on usage, see sphx_glr_auto_examples_impute_plot_missing_values.py.

Trevor Hastie, Robert Tibshirani, David Botstein and Russ B. Altman, Missing value estimation methods for DNA microarrays, BIOINFORMATICS Vol. 17 no. 6, 2001 Pages 520-525. <https://academic.oup.com/bioinformatics/article/17/6/520/272365>`_

Keeping the number of features constant

By default, the scikit-learn imputers will drop fully empty features, i.e. columns containing only missing values. For instance::

توجه: >> imputer = SimpleImputer() >> X = np.array([[np.nan, 1], [np.nan, 2], [np.nan, 3]]) >> imputer.fit_transform(X)

array([[1.], [2.], [3.]])

The first feature in X containing only np.nan was dropped after the imputation. While this feature will not help in predictive setting, dropping the columns will change the shape of X which could be problematic when using imputers in a more complex machine-learning pipeline. The parameter keep_empty_features offers the option to keep the empty features by imputing with a constant value. In most of the cases, this constant value is zero::

توجه: >> imputer.set_params(keep_empty_features=True)

SimpleImputer(keep_empty_features=True)

توجه: >> imputer.fit_transform(X)

array([[0., 1.], [0., 2.], [0., 3.]])

Marking imputed values

The MissingIndicator transformer is useful to transform a dataset into corresponding binary matrix indicating the presence of missing values in the dataset. This transformation is useful in conjunction with imputation. When using imputation, preserving the information about which values had been missing can be informative. Note that both the SimpleImputer and IterativeImputer have the boolean parameter `add_indicator (False by default) which when set to True provides a convenient way of stacking the output of the MissingIndicator` transformer with the output of the imputer.

`NaN is usually used as the placeholder for missing values. However, it enforces the data type to be float. The parameter missing_values allows to specify other placeholder such as integer. In the following example, we will use -1` as missing values::

توجه: >> from sklearn.impute import MissingIndicator >> X = np.array([[-1, -1, 1, 3],

... [4, -1, 0, -1], ... [8, -1, 1, 0]])

توجه: >> indicator = MissingIndicator(missing_values=-1) >> mask_missing_values_only = indicator.fit_transform(X) >> mask_missing_values_only

array([[ True, True, False], [False, True, True], [False, True, False]])

The `features parameter is used to choose the features for which the mask is constructed. By default, it is 'missing-only' which returns the imputer mask of the features containing missing values at fit` time::

توجه: >> indicator.features_

array([0, 1, 3])

The `features parameter can be set to 'all'` to return all features whether or not they contain missing values::

توجه: >> indicator = MissingIndicator(missing_values=-1, features="all") >> mask_all = indicator.fit_transform(X) >> mask_all

array([[ True, True, False, False], [False, True, False, True], [False, True, False, False]])

توجه: >> indicator.features_

array([0, 1, 2, 3])

When using the MissingIndicator in a ~sklearn.pipeline.Pipeline, be sure to use the ~sklearn.pipeline.FeatureUnion or ~sklearn.compose.ColumnTransformer to add the indicator features to the regular features. First we obtain the iris dataset, and add some missing values to it.

توجه: >> from sklearn.datasets import load_iris >> from sklearn.impute import SimpleImputer, MissingIndicator >> from sklearn.model_selection import train_test_split >> from sklearn.pipeline import FeatureUnion, make_pipeline >> from sklearn.tree import DecisionTreeClassifier >> X, y = load_iris(return_X_y=True) >> mask = np.random.randint(0, 2, size=X.shape).astype(bool) >> X[mask] = np.nan >> X_train, X_test, y_train, _ = train_test_split(X, y, test_size=100,

... random_state=0)

Now we create a ~sklearn.pipeline.FeatureUnion. All features will be imputed using SimpleImputer, in order to enable classifiers to work with this data. Additionally, it adds the indicator variables from MissingIndicator.

توجه: >> transformer = FeatureUnion(

... transformer_list=[ ... ('features', SimpleImputer(strategy='mean')), ... ('indicators', MissingIndicator())])

توجه: >> transformer = transformer.fit(X_train, y_train) >> results = transformer.transform(X_test) >> results.shape

(100, 8)

Of course, we cannot use the transformer to make any predictions. We should wrap this in a ~sklearn.pipeline.Pipeline with a classifier (e.g., a ~sklearn.tree.DecisionTreeClassifier) to be able to make predictions.

توجه: >> clf = make_pipeline(transformer, DecisionTreeClassifier()) >> clf = clf.fit(X_train, y_train) >> results = clf.predict(X_test) >> results.shape

(100,)

Estimators that handle NaN values

Some estimators are designed to handle NaN values without preprocessing. Below is the list of these estimators, classified by type (cluster, regressor, classifier, transform):