### A simple study on how to check the statistical goodness of a regression model.

### Photo by Antoine Dautry on Unsplash

**Regression models **are very useful and widely used in machine learning. However, they might show some problems when comes to measure the **goodness **of a trained model. While classification models have some **standard tools** that can be used to assess their performance (i.e. area under the ROC curve, confusion matrix, F-1 score etc.), regression models’ performance can be measured in many different ways. In this article, I’ll show you some techniques I’ve used in my experience as a Data Scientist.

# Example in R

In this example, I’ll show you how to measure the goodness of a trained model using the famous **iris dataset**. I’ll use a **linear regression model** to predict the value of the Sepal Length as a function of the other variables.

First, we’ll load the iris dataset and split it in training and holdout.

set.seed(1)training_idx = sample(1:nrow(iris),nrow(iris)*0.8,replace=FALSE)

holdout_idx = setdiff(1:nrow(iris),training_idx)training = iris[training_idx,]

holdout = iris[holdout_idx,]

Then we can perform a simple **linear regression** in order to describe the variable **Sepal.Length** as a linear function of the others. This is the model we want to check the goodness of.

m = lm(Sepal.Length ~ .,training)

All we need to do now is compare the **residuals **in the training set with the residuals in the holdout. Remember that the residuals are the **differences **between the real value and the predicted value.

holdout_res = holdout$Sepal.Length – predict(m,holdout)

If our training procedure has produced **overfitting**, the residuals in the training set will be **very small** compared with the residuals in the holdout. That’s a negative signal that should invite us to **simplify **the model or **remove **some variables.

Let’s now perform some statistical checks.

## t-test

The first thing we have to check is whether the residuals are biased or not. We know from elementary statistics that the **mean value** of the residuals is **zero**, so we can start checking with a **Student’s t-test **if it’s true or not for our holdout sample.

t.test(holdout_res,mu=0)

As we can see, the p-value is greater than 5%, so we **cannot reject** the null hypothesis and can say that the mean value of the holdout residuals is statistically similar to 0.

Then, we can test if the holdout residuals have the **same average** as the training ones. This is called **Welch’s t-test**.

t.test(training_res,holdout_res)

Again, a p-value higher than 5% can make us tell that **there aren’t enough reasons** to assume that the mean values are different.

## F-test

After we have checked the mean value, there comes the **variance**. We obviously want that the holdout residuals show a behavior **not so much different** from the training residuals, so we can **compare **the variances of the two sets and check whether the holdout variance is higher than the training variance.

A good test to check if a variance is greater than another one is the **F-test**, but it only works with **normally distributed residuals**. If the distribution is not normal, the test might give wrong results.

So, if we really want to use this test, we must check the normality of the residuals using (for example) a **Shapiro-Wilk** test.

Both p-values are higher than 5%, so we can say that both sets **show normally distributed residuals**. We can safely go on performing the F-test.

var.test(training_res,holdout_res)

The p-value is 72%, which is greater than 5% and allows us to say that the two sets have the same variance.

## Kolmogorov-Smirnov test

KS test is very general and useful for many situations. Generally speaking, we expect that, if our model works well, the **probability distribution** of the holdout residuals is **similar **to the probability distribution of the training residuals. The KS test has been created to compare probability distributions, so it can be used for this purpose. However, it carries some approximations that can be dangerous to our analysis. Significative differences between probability distributions can be hidden in the general considerations made by the test. Last, KS distribution is known only with some kind of approximation and, consequently, the p-value; so I suggest to use this test with care.

ks.test(training_res,holdout_res)

Again, the large p-value can make us tell that the two distributions are the same.

## Plot

A Professor of mine at the University usually said: “you have to look at data **by your eyes**”. In machine learning, it’s definitely true.

The best way to take a look at a regression data is by **plotting **the predicted values against the real values in the holdout set. In a perfect condition, we expect that the points lie on the **45 degrees line** passing through the origin (*y = x *is the equation). The nearer the points to this line, the better the regression. If our data make a shapeless blob in the Cartesian plane, there is definitely something wrong.

abline(0,1)

Well, it could have been better, but it’s not completely wrong. Points lie approximatively on the straight line.

## t-test on plot

Finally, we can calculate a linear regression line from the previous plot and check if its intercept is statistically different from zero and its slope is statistically different from 1. To perform these checks, we can use a simple linear model and the statistical theory behind the Student’s t-test.

Remember the definition of the *t* variable with *n-1* degrees of freedom:

When we use the *summarize* function of R on a linear model, it gives us the estimates of the parameters and their standard errors (i.e. the complete denominator of the *t* definition).

For the intercept, we have *mu = 0*, while the slope has *mu = 1*.

s = summary(test_model)intercept = s$coefficients[“(Intercept)”,”Estimate”]

intercept_error = s$coefficients[“(Intercept)”,”Std. Error”]

slope = s$coefficients[“predicted”,”Estimate”]

slope_error = s$coefficients[“predicted”,”Std. Error”]t_intercept = intercept/intercept_errort_slope = (slope-1)/slope_error

Now we have the *t* values, so we can perform a two-sided t-test in order to calculate the p-values.

They are greater than 5% but not too high in absolute value.

# Which method is the best one?

As usual, it depends on the problem. If the residuals are **normally distributed**, t-test and F-test are enough. If they are not, maybe a first plot can help us discover a **macroscopic bias** before using a Kolmogorov-Smirnov test.

However, non-normally distributed residuals should always **raise an alarm** in our head and make us search for some **hidden phenomenon** we haven’t considered yet.

# Conclusions

In this short article, I’ve shown you some methods to calculate the goodness of a regression model. Though there are many possible ways to measure it, these simple techniques can be very useful in many situations and easily explainable to a non-technical audience.