{"id":2319,"date":"2020-03-16T03:46:07","date_gmt":"2020-03-16T00:46:07","guid":{"rendered":"http:\/\/kusuaks7\/?p=1924"},"modified":"2023-12-29T10:20:18","modified_gmt":"2023-12-29T10:20:18","slug":"how-to-measure-the-goodness-of-a-regression-model","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/how-to-measure-the-goodness-of-a-regression-model\/","title":{"rendered":"How To Measure The Goodness Of A Regression Model"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
\n

A simple study on how to check the statistical goodness of a regression model.<\/h2>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

\"How<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Photo by\u00a0Antoine Dautry<\/a>\u00a0on\u00a0<\/span>Unsplash<\/a><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Regression models\u00a0<\/strong>are very useful and widely used in machine learning. However, they might show some problems when comes to measure the\u00a0goodness\u00a0<\/strong>of a trained model. While classification models have some\u00a0standard tools<\/strong>\u00a0that can be used to assess their performance (i.e. area under the ROC curve, confusion matrix, F-1 score etc.), regression models\u2019 performance can be measured in many different ways. In this article, I\u2019ll show you some techniques I\u2019ve used in my experience as a Data Scientist.<\/p>\n<\/section>\n

\n
\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Example in R<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

In this example, I\u2019ll show you how to measure the goodness of a trained model using the famous\u00a0iris dataset<\/strong>. I\u2019ll use a\u00a0linear regression model<\/strong>\u00a0to predict the value of the Sepal Length as a function of the other variables.<\/p>\n

First, we\u2019ll load the iris dataset and split it in training and holdout.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
data(iris)
set.seed(1)training_idx = sample(1:nrow(iris),nrow(iris)*0.8,replace=FALSE)
holdout_idx = setdiff(1:nrow(iris),training_idx)training = iris[training_idx,]
holdout = iris[holdout_idx,]<\/span><\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u00a0<\/p>\n

Then we can perform a simple\u00a0linear regression<\/strong>\u00a0in order to describe the variable\u00a0Sepal.Length<\/strong>\u00a0as a linear function of the others. This is the model we want to check the goodness of.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
m = lm(Sepal.Length ~ .,training)<\/pre>\n
All we need to do now is compare the\u00a0residuals\u00a0<\/strong>in the training set with the residuals in the holdout. Remember that the residuals are the\u00a0differences\u00a0<\/strong>between the real value and the predicted value.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
training_res = training$Sepal.Length – predict(m,training)
holdout_res = holdout$Sepal.Length – predict(m,holdout)<\/span><\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
\u00a0<\/p>\n
If our training procedure has produced\u00a0overfitting<\/strong>, the residuals in the training set will be\u00a0very small<\/strong>\u00a0compared with the residuals in the holdout. That\u2019s a negative signal that should invite us to\u00a0simplify\u00a0<\/strong>the model or\u00a0remove\u00a0<\/strong>some variables.<\/p>\n
Let\u2019s now perform some statistical checks.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
t-test<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
The first thing we have to check is whether the residuals are biased or not. We know from elementary statistics that the\u00a0mean value<\/strong>\u00a0of the residuals is\u00a0zero<\/strong>, so we can start checking with a\u00a0Student\u2019s t-test\u00a0<\/strong>if it\u2019s true or not for our holdout sample.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
t.test(holdout_res,mu=0)<\/pre>\n
\"How<\/p>\n
As we can see, the p-value is greater than 5%, so we\u00a0cannot reject<\/strong>\u00a0the null hypothesis<\/a> and can say that the mean value of the holdout residuals is statistically similar to 0.<\/p>\n
Then, we can test if the holdout residuals have the\u00a0same average<\/strong>\u00a0as the training ones. This is called\u00a0Welch\u2019s t-test<\/strong>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
t.test(training_res,holdout_res)<\/pre>\n
\"How<\/p>\n
Again, a p-value higher than 5% can make us tell that\u00a0there aren\u2019t enough reasons<\/strong>\u00a0to assume that the mean values are different.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
F-test<\/h3>`<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
After we have checked the mean value, there comes the\u00a0variance<\/strong>. We obviously want that the holdout residuals show a behavior\u00a0not so much different<\/strong>\u00a0from the training residuals, so we can\u00a0compare\u00a0<\/strong>the variances of the two sets and check whether the holdout variance is higher than the training variance.<\/p>\n
A good test to check if a variance is greater than another one is the\u00a0F-test<\/strong>, but it only works with\u00a0normally distributed residuals<\/strong>. If the distribution is not normal, the test might give wrong results.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
So, if we really want to use this test, we must check the normality of the residuals using (for example) a\u00a0Shapiro-Wilk<\/strong>\u00a0test.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
\"How<\/p>\n
Both p-values are higher than 5%, so we can say that both sets\u00a0show normally distributed residuals<\/strong>. We can safely go on performing the F-test.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
var.test(training_res,holdout_res)<\/pre>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
The p-value is 72%, which is greater than 5% and allows us to say that the two sets have the same variance.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Kolmogorov-Smirnov test<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
KS test is very general and useful for many situations. Generally speaking, we expect that, if our model works well, the\u00a0probability distribution<\/strong>\u00a0of the holdout residuals is\u00a0similar\u00a0<\/strong>to the probability distribution of the training residuals. The KS test has been created to compare probability distributions, so it can be used for this purpose. However, it carries some approximations that can be dangerous to our analysis. Significative differences between probability distributions can be hidden in the general considerations made by the test. Last, KS distribution is known only with some kind of approximation and, consequently, the p-value; so I suggest to use this test with care.<\/p>\n
ks.test(training_res,holdout_res)<\/pre>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Again, the large p-value can make us tell that the two distributions are the same.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Plot<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
A Professor of mine at the University usually said: \u201cyou have to look at data\u00a0by your eyes<\/strong>\u201d. In machine learning, it\u2019s definitely true.<\/p>\n
\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
The best way to take a look at a regression data is by\u00a0plotting\u00a0<\/strong>the predicted values against the real values in the holdout set. In a perfect condition, we expect that the points lie on the\u00a045 degrees line<\/strong>\u00a0passing through the origin (y = x\u00a0<\/em>is the equation). The nearer the points to this line, the better the regression. If our data make a shapeless blob in the Cartesian plane, there is definitely something wrong.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
plot(holdout$Sepal.Length,predict(m,holdout))
abline(0,1)<\/span><\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Well, it could have been better, but it\u2019s not completely wrong. Points lie approximatively on the straight line.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
t-test on plot<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Finally, we can calculate a linear regression line from the previous plot and check if its intercept is statistically different from zero and its slope is statistically different from 1. To perform these checks, we can use a simple linear model and the statistical theory behind the Student\u2019s t-test.<\/p>\n
Remember the definition of the\u00a0t<\/em>\u00a0variable with\u00a0n-1<\/em>\u00a0degrees of freedom:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
When we use the\u00a0summarize<\/em>\u00a0function of R on a linear model, it gives us the estimates of the parameters and their standard errors (i.e. the complete denominator of the\u00a0t<\/em>\u00a0definition).<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
For the intercept, we have\u00a0mu = 0<\/em>, while the slope has\u00a0mu = 1<\/em>.<\/p>\n
test_model = lm(real ~ predicted, data.frame(real=holdout$Sepal.Length,predicted=predict(m,holdout)))
s = summary(test_model)intercept =\u00a0 s$coefficients[“(Intercept)”,”Estimate”]
intercept_error = s$coefficients[“(Intercept)”,”Std. Error”]
slope = s$coefficients[“predicted”,”Estimate”]
slope_error = s$coefficients[“predicted”,”Std. Error”]t_intercept = intercept\/intercept_errort_slope = (slope-1)\/slope_error<\/span><\/div>\n
\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Now we have the\u00a0t<\/em>\u00a0values, so we can perform a two-sided t-test in order to calculate the p-values.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
They are greater than 5% but not too high in absolute value.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Which method is the best one?<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
As usual, it depends on the problem. If the residuals are\u00a0normally distributed<\/strong>, t-test and F-test are enough. If they are not, maybe a first plot can help us discover a\u00a0macroscopic bias<\/strong>\u00a0before using a Kolmogorov-Smirnov test.<\/p>\n
However, non-normally distributed residuals should always\u00a0raise an alarm<\/strong>\u00a0in our head and make us search for some\u00a0hidden phenomenon<\/strong>\u00a0we haven\u2019t considered yet.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Conclusions<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
In this short article, I\u2019ve shown you some methods to calculate the goodness of a regression model. Though there are many possible ways to measure it, these simple techniques can be very useful in many situations and easily explainable to a non-technical audience.<\/p>\n<\/section>\n\n
 <\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"
Regression models are very useful and widely used in machine learning. However, they might show some problems when comes to measure the goodness of a trained model. Their performance can be measured in many different ways. This article shows you some methods to calculate the goodness of a regression model. Though there are many possible ways to measure it, these simple techniques can be very useful in many situations and easily explainable to a non-technical audience.<\/p>\n","protected":false},"author":618,"featured_media":3984,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[92],"ppma_author":[3328],"class_list":["post-2319","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-machine-learning"],"authors":[{"term_id":3328,"user_id":618,"is_guest":0,"slug":"gianluca-malato","display_name":"Gianluca Malato","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2020\/04\/medium_918623b2-8f36-4110-8343-6fc9228595dd-150x150.jpg","user_url":"http:\/\/www.gianlucamalato.it\/","last_name":"Malato","first_name":"Gianluca","job_title":"","description":"Gianluca Malato is Data Scientist at Poste Italiane SPA.\u00a0 He is also a fiction author and software developer, Editor of\u00a0Data Science Journal<\/a>,\u00a0The Trading Scientist<\/a>, and\u00a0The Writer\u2019s Notebook<\/a>. His books are available on Amazon<\/a>."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2319","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/618"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=2319"}],"version-history":[{"count":6,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2319\/revisions"}],"predecessor-version":[{"id":35237,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2319\/revisions\/35237"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/3984"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=2319"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=2319"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=2319"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=2319"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}