{"id":997,"date":"2018-11-22T02:23:47","date_gmt":"2018-11-21T23:23:47","guid":{"rendered":"http:\/\/kusuaks7\/?p=602"},"modified":"2021-05-17T20:39:37","modified_gmt":"2021-05-17T20:39:37","slug":"train-test-split-and-cross-validation-in-python","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/train-test-split-and-cross-validation-in-python\/","title":{"rendered":"Train\/Test Split and Cross Validation in Python"},"content":{"rendered":"

Ready to learn Data Science? Browse Data Science Training and Certification<\/a> courses developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/em><\/strong><\/p>\n

\n

This post is about Train\/Test Split and Cross Validation. As usual, I am going to give a short overview on the topic and then give an example on implementing it in Python. These are two rather important concepts in data science and data analysis and are used as tools to prevent (or at least minimize) overfitting<\/a>. I\u2019ll explain what that is \u2014 when we\u2019re using a statistical model (like linear regression, for example), we usually fit the model on a training set in order to make predications on a data that wasn\u2019t trained (general data). Overfitting means that what we\u2019ve fit the model too much to the training data. It will all make sense pretty soon, I promise!<\/p>\n

What is Overfitting\/Underfitting a Model?<\/h3>\n

As mentioned, in statistics and machine learning we usually split our data into two subsets: training data and testing data (and sometimes to three: train, validate and test), and fit our model on the train data, in order to make predictions on the test data. When we do that, one of two thing might happen: we overfit our model or we underfit our model. We don\u2019t want any of these things to happen, because they affect the predictability of our model \u2014 we might be using a model that has lower accuracy and\/or is ungeneralized (meaning you can\u2019t generalize your predictions on other data). Let\u2019s see what under and overfitting actually mean:<\/p>\n

Overfitting<\/h4>\n

Overfitting means that model we trained has trained \u201ctoo well\u201d and is now, well, fit too closely to the training dataset. This usually happens when the model is too complex (i.e. too many features\/variables compared to the number of observations). This model will be very accurate on the training data but will probably be very not accurate on untrained or new data. It is because this model is not generalized (or not AS generalized), meaning you can generalize the results and can\u2019t make any inferences on other data, which is, ultimately, what you are trying to do. Basically, when this happens, the model learns or describes the \u201cnoise\u201d in the training data instead of the actual relationships between variables in the data. This noise, obviously, isn\u2019t part in of any new dataset, and cannot be applied to it.<\/p>\n

Underfitting<\/h4>\n

In contrast to overfitting, when a model is underfitted, it means that the model does not fit the training data and therefore misses the trends in the data. It also means the model cannot be generalized to new data. As you probably guessed (or figured out!), this is usually the result of a very simple model (not enough predictors\/independent variables). It could also happen when, for example, we fit a linear model (like linear regression<\/a>) to data that is not linear. It almost goes without saying that this model will have poor predictive ability (on training data and can\u2019t be generalized to other data).<\/p>\n

<\/canvas><\/figure>\n

An example of overfitting, underfitting and a model that\u2019s \u201cjust right!\u201d<\/p>\n

It is worth noting the underfitting is not as prevalent as overfitting. Nevertheless, we want to avoid both of those problems in data analysis. You might say we are trying to find the middle ground between under and overfitting our model. As you will see, train\/test split and cross validation help to avoid overfitting more than underfitting. Let\u2019s dive into both of them!<\/p>\n

Train\/Test Split<\/h3>\n

As I said before, the data we use is usually split into training data and test data. The training set contains a known output and the model learns on this data in order to be generalized to other data later on. We have the test dataset (or subset) in order to test our model\u2019s prediction on this subset.<\/p>\n

<\/canvas><\/figure>\n

Train\/Test Split<\/p>\n

Let\u2019s see how to do this in Python. We\u2019ll do this using the Scikit-Learn library<\/a>and specifically the train_test_split method<\/a>. We\u2019ll start with importing the necessary libraries:<\/p>\n

import pandas as <\/span>pd<\/span>
\nfrom<\/span>sklearn<\/span> import datasets, linear_model
\nfrom sklearn.model_selection import train_test_split
\nfrom<\/span>matplotlib<\/span> import <\/span>pyplot<\/span> as <\/span>plt<\/span><\/div>\n
 <\/div>\n

Let\u2019s quickly go over the libraries I\u2019ve imported:<\/p>\n