Add Binary Flags for Missing Values for Machine Learning

Missing values can cause problems when modeling classification and regression prediction problems with machine learning algorithms.

A common approach is to replace missing values with a calculated statistic, such as the mean of the column. This allows the dataset to be modeled as per normal but gives no indication to the model that the row original contained missing values.

One approach to address this issue is to include additional binary flag input features that indicate whether a row or a column contained a missing value that was imputed. This additional information may or may not be helpful to the model in predicting the target value.

In this tutorial, you will discover how to add binary flags for missing values for modeling.

After completing this tutorial, you will know:

• How to load and evaluate models with statistical imputation on a classification dataset with missing values.
• How to add a flag that indicates if a row has one more missing values and evaluate models with this new feature.
• How to add a flag for each input variable that has missing values and evaluate models with these new features.

Discover data cleaning, feature selection, data transforms, dimensionality reduction and much more in my new book, with 30 step-by-step tutorials and full Python source code.

Let’s get started.

Tutorial Overview

This tutorial is divided into three parts; they are:

1. Imputing the Horse Colic Dataset
2. Model With a Binary Flag for Missing Values
3. Model With Indicators of All Missing Values

Imputing the Horse Colic Dataset

The horse colic dataset describes medical characteristics of horses with colic and whether they lived or died.

There are 300 rows and 26 input variables with one output variable. It is a binary classification prediction task that involves predicting 1 if the horse lived and 2 if the horse died.

There are many fields we could select to predict in this dataset. In this case, we will predict whether the problem was surgical or not (column index 23), making it a binary classification problem.

The dataset has numerous missing values for many of the columns where each missing value is marked with a question mark character (“?”).

Below provides an example of rows from the dataset with marked missing values.

You can learn more about the dataset here:

• Horse Colic Dataset
• Horse Colic Dataset Description

No need to download the dataset as we will download it automatically in the worked examples.

Marking missing values with a NaN (not a number) value in a loaded dataset using Python is a best practice.

We can load the dataset using the read_csv() Pandas function and specify the “na_values” to load values of ‘?’ as missing, marked with a NaN value.

The example below downloads the dataset, marks “?” values as NaN (missing) and summarizes the shape of the dataset.

Running the example downloads the dataset and reports the number of rows and columns, matching our expectations.

Next, we can evaluate a model on this dataset.

We can use the SimpleImputer class to perform statistical imputation and replace the missing values with the mean of each column. We can then fit a random forest model on the dataset.

For more on how to use the SimpleImputer class, see the tutorial:

• Statistical Imputation for Missing Values in Machine Learning

To achieve this, we will define a pipeline that first performs imputation, then fits the model and evaluates this modeling pipeline using repeated stratified k-fold cross-validation with three repeats and 10 folds.

The complete example is listed below.

Running the example evaluates the random forest with mean statistical imputation on the horse colic dataset.

Your specific results may vary given the stochastic nature of the learning algorithm, the stochastic nature of the evaluation procedure, and differences in precision across machines. Try running the example a few times.

In this case, the pipeline achieved an estimated classification accuracy of about 86.2 percent.

Next, let’s see if we can improve the performance of the model by providing more information about missing values.

Model With a Binary Flag for Missing Values

In the previous section, we replaced missing values with a calculated statistic.

The model is unaware that missing values were replaced.

It is possible that knowledge of whether a row contains a missing value or not will be useful to the model when making a prediction.

One approach to exposing the model to this knowledge is by providing an additional column that is a binary flag indicating whether the row had a missing value or not.

• 0: Row does not contain a missing value.
• 1: Row contains a missing value (which was/will be imputed).

This can be achieved directly on the loaded dataset. First, we can sum the values for each row to create a new column where if the row contains at least one NaN, then the sum will be a NaN.

We can then mark all values in the new column as 1 if they contain a NaN, or 0 otherwise.

Finally, we can add this column to the loaded dataset.

Tying this together, the complete example of adding a binary flag to indicate one or more missing values in each row is listed below.

Running the example first downloads the dataset and reports the number of rows and columns, as expected.

Then the new binary variable indicating whether a row contains a missing value is created and added to the end of the input variables. The shape of the input data is then reported, confirming the addition of the feature, from 27 to 28 columns.

We can then evaluate the model as we did in the previous section with the additional binary flag and see if it impacts model performance.

The complete example is listed below.

Running the example reports the mean and standard deviation classification accuracy on the horse colic dataset with the additional feature and imputation.

Your specific results may vary given the stochastic nature of the learning algorithm, the stochastic nature of the evaluation procedure, and differences in precision across machines. Try running the example a few times.

In this case, we see a modest lift in performance from 86.2 percent to 86.3 percent. The difference is small and may not be statistically significant.

Most rows in this dataset have a missing value, and this approach might be more beneficial on datasets with fewer missing values.

Next, let’s see if we can provide even more information about the missing values to the model.

Model With Indicators of All Missing Values

In the previous section, we added one additional column to indicate whether a row contains a missing value or not.

One step further is to indicate whether each input value was missing and imputed or not. This effectively adds one additional column for each input variable that contains missing values and may offer benefit to the model.

This can be achieved by setting the “add_indicator” argument to True when defining the SimpleImputer instance.

We can demonstrate this with a worked example.

The example below loads the horse colic dataset as before, then imputes the missing values on the entire dataset and adds indicators variables for each input variable that has missing values

Running the example first downloads and summarizes the shape of the dataset as expected, then applies the imputation and adds the binary (1 and 0 values) columns indicating whether each row contains a missing value for a given input variable.

We can see that the number of input variables has increased from 27 to 48, indicating the addition of 21 binary input variables, and in turn, that 21 of the 27 input variables must contain at least one missing value.

Next, we can evaluate the model with this additional information.

The complete example below demonstrates this.

Running the example reports the mean and standard deviation classification accuracy on the horse colic dataset with the additional indicators features and imputation.

Your specific results may vary given the stochastic nature of the learning algorithm, the stochastic nature of the evaluation procedure, and differences in precision across machines. Try running the example a few times.

In this case, we see a nice lift in performance from 86.3 percent in the previous section to 86.7 percent.

This may provide strong evidence that adding one flag per column that was inputted is a better strategy on this dataset and chosen model.

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Related Tutorials

• Best Results for Standard Machine Learning Datasets
• Statistical Imputation for Missing Values in Machine Learning
• How to Handle Missing Data with Python

Summary

In this tutorial, you discovered how to add binary flags for missing values for modeling.

Specifically, you learned:

• How to load and evaluate models with statistical imputation on a classification dataset with missing values.
• How to add a flag that indicates if a row has one more missing values and evaluate models with this new feature.
• How to add a flag for each input variable that has missing values and evaluate models with these new features.

Do you have any questions?


Ask your questions in the comments below and I will do my best to answer.