{"id":2023,"date":"2019-10-22T03:56:41","date_gmt":"2019-10-22T03:56:41","guid":{"rendered":"http:\/\/kusuaks7\/?p=1628"},"modified":"2024-03-11T10:39:24","modified_gmt":"2024-03-11T10:39:24","slug":"how-to-measure-feature-importance-in-a-binary-classification-model","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/how-to-measure-feature-importance-in-a-binary-classification-model\/","title":{"rendered":"How to measure feature importance in a binary classification model"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

An example in R language of how to check feature relevance in a binary classification problem<\/strong><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
\n

One of the main tasks that a data scientist must face when he builds a machine learning model is the selection of the\u00a0most predictive variables<\/strong>. Selecting predictors with low predictive power can lead, in fact, to overfitting or\u00a0low model performance<\/strong>. In this article, I\u2019ll show you some techniques to better select the predictors of a dataset in a\u00a0binary classification<\/strong>\u00a0model.<\/p>\n\n<\/section>

\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

When a data scientist starts working at some model, he often\u00a0doesn\u2019t have\u00a0<\/strong>a real idea of which the predictors should be. Maybe the previous phase of\u00a0business understanding<\/strong>\u00a0discarded some useless variables but we often have to face a giant table of\u00a0hundreds<\/strong>\u00a0of variables.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Training a model on such a huge table\u00a0is not a good idea<\/strong>. You really run the risk of\u00a0collinearity\u00a0<\/strong>(i.e. correlations between variables). So we have to\u00a0choose\u00a0<\/strong>the best set of variables to use in order to make our model\u00a0learn properly\u00a0<\/strong>from the business\u00a0information\u00a0<\/strong>we are giving it.<\/p>\n

Our goal is to\u00a0increase\u00a0<\/strong>the predictive power of our model against our binary target, so we must find those variables that are\u00a0strongly correlated\u00a0<\/strong>with it. Remember: information is\u00a0hidden<\/strong>\u00a0inside the dataset and we must provide all the necessary conditions to make our model extract it. So we have to\u00a0prepare<\/strong>\u00a0data\u00a0before<\/strong>\u00a0the training phase in order to make the model work properly.<\/p>\n

Numerical and categorical predictors have a different kind of approach and I\u2019ll show you both.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Numerical variables<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Since our problem is a binary classification task, we can consider our\u00a0outcome\u00a0<\/strong>as a\u00a0number\u00a0<\/strong>which can be either equal to 0 or 1. In order to check if a variable is relevant or not, we can calculate the absolute value of the\u00a0Pearson linear correlation coefficient\u00a0<\/strong>between the target and the predictors.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

That is the covariance divided by the product of the standard deviations.<\/p>\n

We are not interested in the sign of correlation. We just need to know its\u00a0intensity<\/strong>. That\u2019s why we use the absolute value.<\/p>\n

I have often seen this kind of approach in many AI projects and tools. Honestly, I have to say that it\u2019s not completely correct to calculate the correlation coefficient in this way. For a perfect predictor, we expect a Pearson coefficient absolute value equal to 1, but we could not achieve this value if we treat binary outcome as a binary number. It\u2019s not important, however. We are using Pearson correlation coefficient to\u00a0sort our features<\/strong>\u00a0from the most relevant to the least one, so as long as the coefficient calculation is the same, we can\u00a0compare<\/strong>\u00a0the features between them.<\/p>\n

Pearson correlation coefficient is not flawless, however. It only measures\u00a0linear correlation<\/strong>\u00a0and our variables couldn\u2019t be linearly correlated. But in first approximation, we can easily calculate and use it for our purpose.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Categorical variables<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

For the categorical variables, there\u2019s no Pearson correlation coefficient, but we can use another great discovery of Pearson, which is the\u00a0chi-square test<\/strong>.<\/p>\n

Let\u2019s say we have a histogram of\u00a0N<\/em>\u00a0different categories with\u00a0O<\/em>\u00a0observation that sum up to\u00a0n\u00a0<\/em>and let\u2019s say want to compare it with a theoretical histogram made by probabilities\u00a0p<\/em>. We can build a chi-square variable in this way:<\/p>\n

This variable is asymptotically distributed as a\u00a0chi-square distribution<\/strong>\u00a0with\u00a0N<\/em>-1 degrees of freedom.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

This variable is asymptotically distributed as a\u00a0chi-square distribution<\/strong>\u00a0with\u00a0N<\/em>-1 degrees of freedom.<\/p>

If our variable is not correlated to the target, we expect that, for each one of its values, we get 50% zeroes and 50% ones on our dataset. This is a\u00a0theoretical histogram\u00a0<\/strong>we could expect to have if there\u2019s no correlation, so a\u00a0one-tailed chi-square test<\/strong>\u00a0performed to check whether the real histogram is similar to this one, should give us a p-value equal to 1 (i.e. a low chi-square value) if our variable is not correlated to the target. On the contrary, a perfect predictor will push p-value towards\u00a0lower values\u00a0<\/strong>(i.e. higher chi-square values).<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Example in\u00a0R<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

To better explain the procedure I\u2019ll show you an example in R code. I\u2019ll work with the famous\u00a0iris\u00a0<\/strong>dataset.<\/p>\n

Remember that R has a powerful function\u00a0cor<\/strong>\u00a0that calculates the correlation matrix and the function\u00a0chisq.test<\/strong>\u00a0that performs the chi-square test.<\/p>\n

First, we create a column named\u00a0target<\/strong>\u00a0that is equal to 1 when the species is\u00a0virginica<\/strong>\u00a0and 0 otherwise. Then we\u2019ll check the correlations with the other variables.<\/p>\n

Let\u2019s start with the\u00a0numerical features<\/strong>. With this simple code, it\u2019s very easy to find the most correlated ones.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
# Load iris dataset data(\u201ciris\u201d)<\/em><\/div>\n \n
# Generate a binary target column\niris$target = ifelse(iris$Species == \u201cvirginica\u201d,1,0)\nnumeric_columns = setdiff(names(iris),\u201dSpecies\u201d)<\/em><\/span><\/div>\n \n
target_corr = abs(cor(iris[,numeric_columns])[“target\u201d,])<\/span><\/em><\/div>\n

<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAs you can see, the most correlated one is the petal width, then comes the petal length and so on. The correlation of the target with itself is obviously 1.<\/span>\n

Let\u2019s take a look at the plot of the target variable against the\u00a0petal width<\/strong>:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAs you can see, higher values of petal width lead to 1 and lower values lead to 0. That\u2019s a clear correlation.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Now, let\u2019s take a look at the plot of the target against the\u00a0sepal length<\/strong>, which has been classified as the least representative variable:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIt\u2019s clear that there is a wide region approximately between 5.5 and 7 inside which we get 0 and 1 almost alternatively. The lack of a\u00a0graphical pattern<\/strong>\u00a0is always a good reason to suspect the lack of\u00a0correlation<\/strong>.\n

For the\u00a0categorical\u00a0<\/strong>case, we\u2019ll calculate the correlation between the target and the species variables. Of course, we expect a strong correlation, because we have built the target as a direct function of the species.<\/p>\n

I\u2019ll show you the single-line code and the results:<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe\u00a0table<\/strong>\u00a0function generates the contingency table and the\u00a0chisq.test<\/strong>\u00a0function has been built in order to perform the chi-square test as we want for our case.\n

A very low p-value means a very strong difference from the uncorrelated case. As usual in the hypothesis tests, you don\u2019t actually accept the null hypothesis, but\u00a0refuse to neglect<\/strong>\u00a0it.<\/p>\n

We can get further confirmation by taking a look at the contingency table:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAs you can see, the column values are\u00a0very unbalanced<\/strong>, which is exactly what we are looking for.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Conclusions<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

In this article, I\u2019ve shown you two simple techniques in R to measure the importance of numerical and categorical variables against a binary target. There are many more methods that can be used both with a multi-class categorical target and for a numerical target (i.e. regression).<\/p>\n

However, this simple procedure can be used to check at first the most important variables and start a deeper analysis to find the best set of predictors for our model.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

An example in R language of how to check feature relevance in a binary classification problem One of the main tasks that a data scientist must face when he builds a machine learning model is the selection of the\u00a0most predictive variables. Selecting predictors with low predictive power can lead, in fact, to overfitting or\u00a0low model<\/p>\n","protected":false},"author":618,"featured_media":2448,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[92],"ppma_author":[3328],"class_list":["post-2023","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-machine-learning"],"authors":[{"term_id":3328,"user_id":618,"is_guest":0,"slug":"gianluca-malato","display_name":"Gianluca Malato","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2020\/04\/medium_918623b2-8f36-4110-8343-6fc9228595dd-150x150.jpg","user_url":"http:\/\/www.gianlucamalato.it\/","last_name":"Malato","first_name":"Gianluca","job_title":"","description":"Gianluca Malato is Data Scientist at Poste Italiane SPA.\u00a0 He is also a fiction author and software developer, Editor of\u00a0Data Science Journal<\/a>,\u00a0The Trading Scientist<\/a>, and\u00a0The Writer\u2019s Notebook<\/a>. His books are available on Amazon<\/a>."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2023","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/618"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=2023"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2023\/revisions"}],"predecessor-version":[{"id":36330,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2023\/revisions\/36330"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/2448"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=2023"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=2023"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=2023"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=2023"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}