{"id":2028,"date":"2019-10-24T02:24:46","date_gmt":"2019-10-24T02:24:46","guid":{"rendered":"http:\/\/kusuaks7\/?p=1633"},"modified":"2024-03-08T13:03:15","modified_gmt":"2024-03-08T13:03:15","slug":"feature-selection-by-random-search-in-python","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/feature-selection-by-random-search-in-python\/","title":{"rendered":"Feature selection by random search in Python"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tFeature selection<\/strong>\u00a0has always been a great task in machine learning. According to my experience, I can surely say that feature selection\u00a0is much more important<\/strong>\u00a0than model selection itself.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Feature selection and collinearity<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

I have already written an article\u00a0about feature selection. It was an\u00a0unsupervised\u00a0<\/strong>way to measure feature importance in a\u00a0binary classification<\/strong>\u00a0model, using Pearson\u2019s chi-square test and correlation coefficient.<\/p>\n

Generally speaking, an unsupervised approach is often enough for a simple feature selection. However, each model has\u00a0its own way<\/strong>\u00a0of \u201cthinking\u201d the features and treat their correlation with the target variable. Moreover, there are models that do not care too much about\u00a0collinearity\u00a0<\/strong>(i.e., the correlation between the features) and other models that show\u00a0very big problems<\/strong>\u00a0when it occurs (for example, linear models).<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Although it\u2019s possible to\u00a0rank\u00a0<\/strong>the features by some kind of\u00a0relevance<\/strong>\u00a0metrics introduced by the model (for example, the p-value of the t-test performed on the coefficients of linear regression), taking only the most relevant variables couldn\u2019t be enough. Think about a feature that is equal to another one, just multiplied by two. The linear correlation between these features if 1 and this simple multiplication doesn\u2019t affect the correlation with the target variable, so if we take only the most relevant variables, we\u2019ll take the original feature and the multiplied one. This leads to\u00a0collinearity<\/strong>, which can be quite dangerous for our model.<\/p>\n

That\u2019s why we must introduce some way to better select our features.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Random search<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Random search is a really useful tool in a data scientist toolbox. It\u2019s a very simple technique very often used, for example, in cross-validation and\u00a0hyperparameter optimization<\/strong>.<\/p>\n

It\u2019s very simple. If you have a multi-dimensional grid and want to look for the point on this grid which maximizes (or minimizes) some\u00a0objective function<\/strong>, random search works as follows:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
    \n \t
  1. Take a random point on the grid and measure the objective function value<\/li>\n \t
  2. If the value is better than the best one achieved so far, keep the point in memory.<\/li>\n \t
  3. Repeat for a certain, pre-defined number of times<\/li>\n<\/ol>\n

    That\u2019s it. Just generating random points and look for the best one.<\/p>\n

    Is this a\u00a0good way<\/strong>\u00a0to find the global minimum (or maximum)? Of course, it\u2019s not. The point we are looking for is only one (if we are lucky) in a very large space and we have only a limited number of iterations. The probability of getting that single point in an\u00a0N-<\/em>point grid is\u00a01\/N<\/em>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    So, why is random search so much used? Because we\u00a0never really want<\/strong>\u00a0to maximize our performance measure; we want a good,\u00a0reasonably high value<\/strong>\u00a0that it\u2019s not the highest possible, to avoid overfitting.<\/p>\n

    That\u2019s why random search works and can be used in feature selection.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t

    How to use random search for feature selection<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Random search can be used for feature selection with quite\u00a0good results<\/strong>. An example of a procedure similar to a random search is the\u00a0Random Forest<\/strong>\u00a0model which performs a random selection of the features for each tree.<\/p>\n

    The idea is pretty simple: choose the features\u00a0randomly<\/strong>, measure the model performances by\u00a0k-fold cross-validation<\/strong>, and repeat many times. The feature combination that gives the best performances is the one we are looking for.<\/p>\n

    More precisely, these are the steps to follow:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
      \n \t
    1. Generate a random integer number\u00a0N<\/em>between 1 and the number of features.<\/li>\n \t
    2. Generate a random sequence of\u00a0N<\/em>integer numbers between 0 and\u00a0N-1<\/em>without repetition. This sequence represents our feature array. Remember that Python arrays start from 0.<\/li>\n \t
    3. Train the model on these features and cross-validate it with k-fold cross-validation, saving the\u00a0average value\u00a0<\/strong>of some performance measure.<\/li>\n \t
    4. Repeat from point 1 as many times as you want.<\/li>\n \t
    5. Finally, get the feature array that gives the best performances according to the chosen performance measure.<\/li>\n<\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t

      A practical example in Python<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t

      For this example, I\u2019ll use the\u00a0breast cancer dataset<\/strong>\u00a0included in\u00a0sklearn<\/strong>\u00a0module. Our model will be a\u00a0logistic regression<\/strong>, and we\u2019ll perform a 5-fold cross-validation using\u00a0accuracy<\/strong>\u00a0as the performance measure.<\/p>\n

      First of all, we must import the necessary modules.<\/p>\n\n

      import sklearn.datasets\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import cross_val_score\nimport numpy as np<\/span><\/div>\n

      Then we can import breast cancer data and break it in input and target.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t
      dataset= sklearn.datasets.load_breast_cancer()\ndata = dataset.data\ntarget = dataset.target<\/span><\/div>\n

      We can now create a logistic regression object.<\/p>\n\n

      lr = LogisticRegression()\n<\/pre>\nThen, we can measure the average accuracy in k-fold CV with all the features.<\/p.\n
      # Model accuracy using all the features\nnp.mean(cross_val_score(lr,data,target,cv=5,scoring=”accuracy”))\n# 0.9509041939207385<\/span><\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\tIt\u2019s 95%. Let\u2019s keep this in mind.\n
      Now, we can implement a random search with, for example, 300 iterations.<\/p>\n\n
      result = []<\/span><\/div>\n
      # Number of iterations\nN_search = 300<\/span><\/div>\n
      # Random seed initialization\nnp.random.seed(1)<\/span><\/div>\n
      for i in range(N_search):\n# Generate a random number of features\nN_columns =\u00a0 list(np.random.choice(range(data.shape[1]),1)+1)<\/span><\/div>\n
      \u00a0\u00a0\u00a0 # Given the number of features, generate features without replacement\ncolumns = list(np.random.choice(range(data.shape[1]), N_columns, replace=False))<\/span><\/div>\n
      \u00a0\u00a0\u00a0 # Perform k-fold cross validation\nscores = cross_val_score(lr,data[:,columns], target, cv=5, scoring=”accuracy”)<\/span><\/div>\n
      \u00a0\u00a0\u00a0 # Store the result\nresult.append({‘columns’:columns,’performance’:np.mean(scores)})<\/span><\/div>\n
      # Sort the result array in descending order for performance measure\nresult.sort(key=lambda x : -x[‘performance’])<\/span><\/div>\n \n
      At the end of the loop and the sorting function, the first element of\u00a0result\u00a0<\/em>list is the object we are looking for.<\/p>\n
      We can use this value to calculate the new performance measure with this subset of the features.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t
      np.mean(cross_val_score(lr, data[:,result[0][\u2018columns\u2019]], target, cv=5, scoring=\u201daccuracy\u201d))\n# 0.9526741054251634<\/span><\/div>\n \n
      As you can see, accuracy has increased.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t
      Conclusions<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t
      Random search can be a powerful tool to perform feature selection. It\u2019s not meant to give the reasons why some features are more useful than other ones (as opposed to other feature selection procedures like Recursive Feature Elimination), but it can be a useful tool to reach\u00a0good results<\/strong>\u00a0in less time.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"
      Random search is a really useful tool in a data scientist toolbox. It’s a very simple technique and can be a powerful tool to perform feature selection. It’s not meant to give the reasons why some features are more useful than other ones (as opposed to other feature selection procedures like Recursive Feature Elimination), but it can be a useful tool to reach good results in less time. Learn how to use a simple random search in Python to get good results in less time.<\/p>\n","protected":false},"author":618,"featured_media":2470,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[94],"ppma_author":[3328],"class_list":["post-2028","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-data-science"],"authors":[{"term_id":3328,"user_id":618,"is_guest":0,"slug":"gianluca-malato","display_name":"Gianluca Malato","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2020\/04\/medium_918623b2-8f36-4110-8343-6fc9228595dd-150x150.jpg","user_url":"http:\/\/www.gianlucamalato.it\/","last_name":"Malato","first_name":"Gianluca","job_title":"","description":"Gianluca Malato is Data Scientist at Poste Italiane SPA.\u00a0 He is also a fiction author and software developer, Editor of\u00a0Data Science Journal<\/a>,\u00a0The Trading Scientist<\/a>, and\u00a0The Writer\u2019s Notebook<\/a>. His books are available on Amazon<\/a>."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2028","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/618"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=2028"}],"version-history":[{"count":5,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2028\/revisions"}],"predecessor-version":[{"id":36313,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2028\/revisions\/36313"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/2470"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=2028"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=2028"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=2028"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=2028"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}