{"id":22556,"date":"2021-01-12T10:01:58","date_gmt":"2021-01-12T10:01:58","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/avoid-mistakes-training-machine-learning-model\/"},"modified":"2023-09-06T13:08:54","modified_gmt":"2023-09-06T13:08:54","slug":"avoid-mistakes-training-machine-learning-model","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/avoid-mistakes-training-machine-learning-model\/","title":{"rendered":"Avoid These 8 Mistakes Before Training A Machine Learning Model"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

One of the most common misconceptions in Machine Learning is that ML Engineers get a CSV dataset and they spend the majority of the time optimizing the hyperparameters of a model.<\/p>\n\n

If you work in the industry, you know that\u2019s far from the truth. ML Engineers spend most of the time planning how to construct the training set that resembles real-world data distribution for a certain problem.<\/strong><\/p>\n\n

When you\u2019ve managed to construct such training set, just add a few well-crafted features and the Machine Learning model won\u2019t have a hard time finding the decision boundary.<\/p>\n\n

In this article, we\u2019re going to go through 8 Machine Learning tips that will help you to train a model with fewer screw-ups. These tips are most useful when you need to construct the training set, e.g. you didn\u2019t get it from Kaggle.<\/p>\n\n

At the end of the article, I also share a link to the Jupyter Notebook template, which you can incorporate into your Machine Learning workflow.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Dataset sample<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Avoid\t\t\t\t\t\t\t\t\t\t\t
Photo by\u00a0Road Trip with Raj<\/a>\u00a0on\u00a0Unsplash<\/a><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

It\u2019s easier to learn with examples. Let\u2019s create a sample dataset with random features.<\/p>\n\n

One row represents a customer with his features and a binary target variable. customer_id is an index in the DataFrame.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
np.<\/strong>random.<\/strong>seed(42)n =<\/strong> 1000
df =<\/strong> pd.<\/strong>DataFrame(
    {
        \"customer_id\": [\"customer_%d\" %<\/strong> i for<\/strong> i in<\/strong> range(n)],
        \"product_a_ratio\": np.<\/strong>random.<\/strong>random_sample(n),
        \"std_price_product_a\": np.<\/strong>random.<\/strong>normal(0, 1, n),
        \"n_purchases_product_a\": np.<\/strong>random.<\/strong>randint(0, 10, n),
    }
)
df.<\/strong>loc[100, \"std_price_product_a\"] =<\/strong> pd.<\/strong>NA
df[\"product_b_ratio\"] =<\/strong> df[\"product_a_ratio\"]
df[\"y\"] =<\/strong> np.<\/strong>random.<\/strong>randint(0, 2, n)
df.<\/strong>set_index(\"customer_id\", inplace=<\/strong>True)df.<\/strong>head()<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Avoid\t\t\t\t\t\t\t\t\t\t\t
Sample dataset with features and target<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
1. Check the target<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Avoid\t\t\t\t\t\t\t\t\t\t\t
Photo by\u00a0Ricardo Arce<\/a>\u00a0on\u00a0Unsplash<\/a><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Despite being obvious that no positive customer is also marked as a negative, it does happen in the real world and it\u2019s worthwhile to check it.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
assert<\/strong> (
    len(set(df[df.<\/strong>y ==<\/strong> 0].<\/strong>index).<\/strong>intersection(df[df.<\/strong>y ==<\/strong> 1])) ==<\/strong> 0
), \"Positive customers have intersection with negative customers\"<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
I use the assert statement above in a Jupyter Notebook, which breaks execution if there is a mistake in the training set. So when I construct a new training set and use the Jupyter command: \u201cRestart kernel and run all cells\u201d, I can be sure that the trainset has the required properties.<\/p>\n\n
When doing Exploratory Data Analysis (EDA), we need to aware that real-world datasets have mistakes in unexpected places. One of the goals of EDA is to discover them.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
2. Check duplicates<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Avoid\t\t\t\t\t\t\t\t\t\t\t
Photo by\u00a0Stefano Ferretti<\/a>\u00a0on\u00a0Unsplash<\/a><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
How do the duplicates come into the training set?<\/p>\n\n
Many times with joins in SQL databases!<\/p>\n\n
E.g. you join an SQL table with another table by customer_id key. If any of those tables have multiples entries for a customer_id, it will create duplicates.<\/p>\n\n
How can we make sure there aren\u2019t any duplicates in our training set?<\/p>\n\n
We can use an assert statement that will break execution in case a duplicated customer_id appears:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
assert<\/strong> len(df[df.<\/strong>index.<\/strong>duplicated()]) ==<\/strong> 0, \"There are duplicates in trainset\"<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
3. Check missing values<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Avoid\t\t\t\t\t\t\t\t\t\t\t
Photo by\u00a0Michael Dziedzic<\/a>\u00a0on\u00a0Unsplash<\/a><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
In my experience, missing values appear for two reasons:<\/strong><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t