{"id":1198,"date":"2019-02-15T10:32:01","date_gmt":"2019-02-15T10:32:01","guid":{"rendered":"http:\/\/kusuaks7\/?p=803"},"modified":"2023-07-17T16:27:02","modified_gmt":"2023-07-17T16:27:02","slug":"working-with-missing-data-in-machine-learning","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/working-with-missing-data-in-machine-learning\/","title":{"rendered":"Working with Missing Data in Machine Learning"},"content":{"rendered":"

Ready to learn Data Science? Browse courses<\/a> like Data Science Training and Certification<\/a> developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/em><\/strong><\/p>\n

Missing values are representative of the messiness of real world data. There can be a multitude of reasons why they occur — ranging from human errors during data entry, incorrect sensor readings, to software bugs in the data processing pipeline.<\/p>\n

The normal reaction is frustration. Missing data are probably the most widespread source of errors in your code, and the reason for most of the exception-handling. If you try to remove them, you might reduce the amount of data you have available dramatically — probably the worst that can happen in machine learning.<\/p>\n

Still, often there are hidden patterns in missing data points. Those patterns can provide additional insight in the problem you’re trying to solve.<\/p>\n

We can treat missing values in data the same way as silence in music — on the surface they might be considered negative (not contributing any information), but inside lies a lot of potential.<\/em><\/p>\n

Methods<\/strong><\/h2>\n

Note: we will be using Python and a <\/em>census data set<\/em><\/a> (modified for the purposes of this tutorial)<\/em><\/p>\n

You might be surprised to find out how many methods for dealing missing data exist. This is a testament to both how important this issue is, and also that there is a lot of potential for creative problem solving.<\/p>\n

The first thing you should do is count how many you have and try to visualize their distributions. For this step to work properly you should manually inspect the data (or at least a subset of it) to try to determine how they are designated. Possible variations are: ‘NaN’, ‘NA’, ‘None’, ‘ ’, ‘?’ and others. If you have something different than ‘NaN’ you should standardize them by using np.nan. To construct our visualizations we will use the handy missingno<\/a>package.<\/p>\n

import missingno as msno
\nmsno.matrix(census_data)<\/p>\n

\"experfy-blog\" <\/p>\n

Missing data visualisation. White fields indicate NA’s<\/p>\n

import pandas as pd
\ncensus_data.isnull().sum()<\/p>\n

age                                  325
\nworkclass                       2143
\nfnlwgt                               325
\neducation                        325
\neducation.num                325
\nmarital.status                   325
\noccupation                     2151
\nrelationship                     326
\nrace                                 326
\nsex                                  326
\ncapital.gain                     326
\ncapital.loss                     326
\nhours.per.week              326
\nnative.country                906
\nincome                           326
\ndtype: int64<\/p>\n

Let’s start with the most simple thing you can do: removal. As mentioned before, while this is a quick solution, and might work in some cases when the proportion of missing values is relatively low (<10%), most of the time it will make you lose a ton of data. Imagine that just because of missing values in one of your features you have to drop the whole observation, even if the rest of the features are perfectly filled and informative!<\/p>\n

import numpy as np
\ncensus_data = census_data.replace('np.nan', 0)<\/p>\n

The second-worst method of doing this is replacement with 0 (or -1). While this would help you run your models, it can be extremely dangerous. The reason for this is that sometimes this value can be misleading. Imagine a regression problem where negative values occur (such as predicting temperature) — well in that case this becomes an actual data point.<\/p>\n

Now that we have those out of the way, let’s become more creative. We can split the type of missing values by their parent datatype:<\/p>\n

Numerical NaNs<\/strong><\/p>\n

A standard and often very good approach is to replace the missing values with mean, median or mode. For numerical values you should go with mean, and if there are some outliers try median (since it is much less sensitive to them).<\/p>\n

from<\/strong> sklearn.preprocessing import<\/strong> Imputer
\nimputer = Imputer(missing_values=np.nan, strategy='median', axis=0)
\ncensus_data[['fnlwgt']] = imputer.fit_transform(census_data[['fnlwgt']])<\/p>\n

Categorical NaNs<\/strong><\/p>\n

Categorical values can be a bit trickier, so you should definitely pay attention to your model performance metrics after editing (compare before and after). The standard thing to do is to replace the missing entry with the most frequent one:<\/p>\n

census_data['marital.status'].value_counts()<\/p>\n

Married-civ-spouse                14808
\nNever-married                       10590
\nDivorced                                  4406
\nSeparated                                 1017
\nWidowed                                   979
\nMarried-spouse-absent            413
\nMarried-AF-spouse                    23
\nName: marital.status, dtype: int64<\/p>\n

def replace_most_common(x):
\n    if pd.isnull(x):
\n        return most_common
\n    else:
\n        return x<\/p>\n

census_data = census_data['marital.status'].map(replace_most_common)<\/p>\n

Conclusion<\/strong><\/h2>\n

The take-home message is that you should be aware of the different methods available to get more out of missing data, and more importantly start regarding it as a source of possible insight instead of annoyance!<\/p>\n

Happy coding \ud83d\ude42<\/p>\n

Bonus — advanced methods and visualizations<\/strong><\/h2>\n

You can theoretically impute missing values by fitting a regression model, such as linear regression or k nearest neighbors. The implementation of this is left as an example to the reader.<\/p>\n

\"experfy-blog\"<\/p>\n

A visual example of kNN.<\/p>\n

Here are some visualisations that are also available from the wonderful missingno<\/a> package, which can help you uncover relationships, in the form of a correlation matrix or a dendrogram:<\/p>\n

\"experfy-blog\" <\/p>\n

Correlation matrix of missing values. Values which are often missing together can help you solve the problem.<\/p>\n

\"experfy-blog\"<\/p>\n

Dendrogram of missing values<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Ready to learn Data Science? Browse courses like Data Science Training and Certification developed by industry thought leaders and Experfy in Harvard Innovation Lab. Missing values are representative of the messiness of real world data. There can be a multitude of reasons why they occur — ranging from human errors during data entry, incorrect sensor readings, to<\/p>\n","protected":false},"author":100,"featured_media":4374,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[94],"ppma_author":[2637],"class_list":["post-1198","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-data-science"],"authors":[{"term_id":2637,"user_id":100,"is_guest":0,"slug":"boyan-angelov","display_name":"Boyan Angelov","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Angelov","first_name":"Boyan","job_title":"","description":"Boyan Angelov leads the machine learning efforts at a Berlin startup, building an AI to help tech companies get better candidates. He has started using machine learning during his work on microbial metagenomes at the Max Planck Institute for Marine Microbiology. The discoveries he made there were in the applications of dimensionality reduction methods. Later he worked in the clinical trials space, focusing on information retrieval and natural language processing.  "}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1198","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/100"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=1198"}],"version-history":[{"count":2,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1198\/revisions"}],"predecessor-version":[{"id":29280,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1198\/revisions\/29280"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/4374"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=1198"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=1198"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=1198"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=1198"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}