As data scientists, one of the key ways we help our collaborators and clients is by sifting through large complex collections of data to help identify the underlying factors associated with a particular outcome (e.g. click-through-rate, conversion rate, predicted revenue).\u00a0In practical terms, the identification of these predictive variables provides insight into how certain components of ad campaigns, customer demographics, etc., are related to outcomes. By having a better understanding of what these factors are, we can more accurately address and predict user needs.<\/p>\n
In statistical\/machine learning parlance this is commonly referred to as variable or model selection. For people who have taken a stats course on linear models or\u00a0searched for methods regarding variable selection, regression\u00a0or classification, you have probably come across some of the staples, such as\u00a0forward, backward and stepwise selection. While these approaches work well in instances when there are a limited number of variables (on the order of ten or less), as we move to tens, hundreds, thousands and more (as in many web-scale applications) they are simply no longer practical nor reliable. From a practical standpoint, filtering out irrelevant variables allows us to focus time and resources on the factors (e.g. ad campaings, user demographics, device our site was accessed from, etc.) that are driving sales, conversion, clicks, and other important metrics.<\/p>\n
I\u2019ll give a quick overview of these procedures, highlight some of their issues and introduce a more \u201cmodern\u201d framework that leverages so-called sparsity constraints that many companies, including the likes of Yahoo and Google, use to wrangle data with many variables. \u00a0For this post I\u2019ll focus on linear regression and show how to implement these various methods in R.<\/p>\n
First recall that given a set of inputs measurements x1<\/sub>,x2<\/sub>,x3<\/sub>….xP<\/sub><\/span><\/span><\/strong>\u00a0(e.g. customer demographics, time of a day user’s logged on, etc.) where xj\u00a0<\/sub>= (x1j<\/sub>,…..,xNj<\/sub>)<\/strong>, the standard linear model is expressed as:<\/p>\n ? =\u00a0\u00df0\u00a0<\/sub>+\u00a0\u00df1<\/sub>x1\u00a0<\/sub>+ \u00df2<\/sub>x2\u00a0<\/sub>+ ……. +\u00a0<\/span>\u00a0\u00dfp<\/sub>xp<\/sub><\/span><\/span><\/strong><\/p>\n where ?<\/strong>\u00a0is some outcome interest (e.g. total sales for a particular user),\u00a0\u00df0<\/sub><\/span><\/strong>\u00a0is the intercept term (representing, for example, average sales) and\u00a0\u00dfj<\/sub><\/strong><\/span>‘s are the variable weights (i.e. coefficients representing that variable’s importance\/contribution to the observed outcome). The criteria by which we estimate the\u00a0\u00dfj<\/sub><\/strong>‘s is minimizing the following loss function with respect to the \u00df<\/span><\/strong>\u00a0:<\/p>\n <\/p>\n Where yi<\/sub><\/strong>‘s are the observed outcomes. The subscript\u00a0i\u00a0<\/em>denotes a single observation (e.g. one users visit onto a site) and\u00a0RSS\u00a0<\/em>stands for “residual sum of squares”.<\/p>\n Here are some of the classical approaches that assess which variables to keep using:<\/p>\n Forward selection<\/em>: Starting with no variables in the model, add one at a time based on which is the most statistically significant. Significance is typically assessed based on one of several criteria; an individual variable\u2019s p-value, a comparison of the model with and without that variable (a comparison of the \u201cfull\u201d versus \u201creduced\u201d model that also results in a p-value) or the use of \u201cinformation criteria\u201d such as AIC and BIC. The algorithm continues adding variables and terminates when none of the remaining variables has a significant p-value (e.g. less than 0.05).<\/p>\n Backward selection<\/em>: Similar to forward selection, except that now we start with all of the variables in the model and remove one variable at a time, which variable is removed is, once again most commonly determined by a p-value. The algorithm terminates when all remaining variables are significant and the non-significant ones have been removed.<\/p>\n Stepwise selection<\/em>: Is a combination of forward and backward procedures. There are number of variants of this algorithm, the most common one follows a forward procedure, but after a new variable has been added one backward selection step is taken to identify if any variables should be removed.<\/p>\n As previously mentioned, while the methods above work sufficiently well for a handful of variables, as we move to larger sets things start getting messy. For example, even at 20 variables, using something like the stepwise approach is not only comparatively slow but also highly unreliable. Consider the fact that we\u2019re only exploring a small subspace of the over 1 million possible models (i.e. all models with 1, 2, 3 variables etc. from 20). This issue is rapidly compounded as we move to larger numbers. What we\u2019d ideally like is an approach that searches this enormous space of models more efficiently.<\/p>\n To make this a little more concrete, consider the following example; a recently organized competition\u00a0asked participants to predict restaurant revenue based on a collection of variables like location, demographics, etc. The data that was provided contained 137 restaurants\u00a0and 42 variables.\u00a0This data was to be used to build a model to predict revenue for 100,000 restaurants for which revenue information was \u201cmasked.\u201d\u00a0Since the actual revenue for those 100,000 is not publicly available, we\u2019ll focus our analysis on the data provided.<\/p>\n The first takeaway is to note the large number of variables compared to the number of\u00a0restaurants (this imbalance is not uncommon in webscale data where while there may be billions of observations, there are also millions of variables). Generally speaking these types of scenarios, if not carefully accounted for,<\/ins> tend to produce models that are highly biased, i.e. do not provide accurate prediction outside the available data (commonly referred to as \u201coverfitting\u201d).<\/p>\n The appeal of model selection procedures like those mentioned above is that they help us avoid overfitting by eliminating variables that are not predictive and end up introducing noise. This also allows for clearer insights into what\u2019s driving revenue. For someone looking into investing in a restaurant this information might be valuable as it would tell them what factors are most critical for\/correlated with success\/growth.<\/p>\nApproaches to Variable Selection<\/h3>\n
Illustration: Restaurant Revenue Prediction<\/h3>\n