{"id":976,"date":"2018-11-12T04:20:29","date_gmt":"2018-11-12T01:20:29","guid":{"rendered":"http:\/\/kusuaks7\/?p=581"},"modified":"2021-05-11T14:01:18","modified_gmt":"2021-05-11T14:01:18","slug":"stop-feeding-garbage-to-your-model-the-6-biggest-mistakes-with-datasets-and-how-to-avoid-them","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/stop-feeding-garbage-to-your-model-the-6-biggest-mistakes-with-datasets-and-how-to-avoid-them\/","title":{"rendered":"Stop Feeding Garbage To Your Model!\u200a\u2014\u200aThe 6 biggest mistakes with datasets and how to avoid them"},"content":{"rendered":"

Ready to learn Big Data? Browse Big Data Training and Certification Courses<\/a> developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/em><\/strong><\/p>\n

\n
\n

Learn how to build killer datasets by avoiding the most frequent mistakes in Data Science, plus tips, tricks and kittens.<\/p>\n<\/blockquote>\n

\n

<\/canvas><\/p>\n<\/figure>\n<\/section>\n

\n
\n

Introduction<\/h3>\n

If you haven’t heard it already, let me tell you a truth that you should, as a data scientist<\/em>, always keep in a corner of your head:<\/p>\n

“Your results are only as good as your data.”<\/strong><\/p><\/blockquote>\n

Many people make the mistake of trying to compensate <\/strong>for <\/strong>their ugly dataset<\/strong>by improving<\/strong> their model<\/strong>. This is the equivalent of buying a supercar<\/strong>because your old car doesn’t perform well with cheap<\/em> gasoline<\/em>. It makes much more sense to refine<\/em> the oil<\/strong> instead of upgrading<\/em> the car<\/strong>. In this article, I will explain how you can easily improve<\/strong> your results<\/strong> by enhancing<\/strong> your dataset<\/strong>.<\/p>\n

Note<\/em><\/strong>: I will take the task of image classification as an example, but these tips can be applied to all sorts of datasets.<\/em><\/p>\n

The 6 Most frequent mistakes, and how to fix them.<\/h3>\n
\n

<\/canvas><\/p>\n<\/figure>\n

1. Not enough data.<\/h4>\n

If your dataset is too small<\/em>, your model doesn’t have enough examples to find discriminative features<\/strong> that will be used to generalize<\/strong>. It will then overfit<\/strong><\/a>your data, resulting in a low<\/strong> training<\/strong> error<\/strong> but a high<\/strong> test<\/strong> error<\/strong>.<\/p>\n

Solution #1: <\/em><\/strong>gather more data. You can try to find more from the same<\/em> source<\/em>as your original dataset, or from another<\/em> source<\/em> if the images are quite similar or if you absolutely want<\/strong> to generalize<\/strong>.<\/p>\n

Caveats: <\/em><\/strong>This is usually not an easy thing to do, at least without investing time and money. Also, you might want to do an analysis<\/em> to determine how much <\/strong>additional <\/strong>data you need. Compare your results with different dataset sizes,<\/strong> and try to extrapolate<\/strong>.<\/p>\n

\n

<\/canvas><\/p>\n<\/figure>\n

In this case, it seems that we would need 500k samples<\/strong> to reach our target<\/strong> error<\/strong>. That would mean gathering 50 times as much<\/strong> data as we have for the moment. It is probably more efficient<\/em> to work on other aspects<\/em><\/strong> of the data, or on the model<\/strong>.<\/p>\n

Solution #2: <\/em><\/strong>augment your data by creating multiple copies of the same image with slight variations. This technique works wonders and it produces tons of additional images at a really low cost. You can try to crop<\/em>, rotate<\/em>, translate<\/em> or scale<\/em> your image. You can add<\/em> noise<\/em>, blur it<\/em>, change its colors <\/em>or obstruct parts of it.<\/em> In all cases, you need to make sure the data is still representing the same class<\/strong><\/p>\n

\n

<\/canvas><\/p>

 <\/figcaption><\/figure>\n

All this images still represent the “cat” category<\/p>\n

This can be extremely powerful, as stacking these effects gives exponentially numerous samples for your dataset. Note that this is still usually inferior<\/strong> to collecting more<\/strong> raw<\/strong> data<\/strong>.<\/p>\n

<\/p>\n

Combined data augmentation techniques. The class is still “cat” and should be recognized as such.<\/p>\n

Caveats: <\/em><\/strong>all augmentations techniques might not be usable for your problem. For example, if you want to classify Lemons and Limes, don’t play with the hue, as it would make sense <\/em>that color is important for the classification.<\/p>\n

\n

<\/canvas><\/p>\n

<\/p>

This type of data augmentation would make it harder for the model to find discriminating features.<\/figcaption><\/figure>\n
\n

<\/canvas><\/p>\n

<\/p>\n<\/figure>\n

2. Low quality classes<\/h4>\n

It’s an easy one, but take time to go through your dataset if possible, and verify the label <\/strong>of each sample. This might take a while, but having counter-examples <\/em>in your dataset will be detrimental <\/strong>to <\/strong>the learning process.<\/p>\n

Also, choose the right level of granularity<\/strong> for your classes. Depending on the problem, you might need more or less classes. For example<\/em>, you can classify the image of a kitten<\/strong> with a global classifier <\/strong>to determine it’s an animal<\/strong>, then run it through an animal classifier <\/strong>to determine it’s a kitten<\/strong>. A huge model could do both, but it would be much harder.<\/p>\n

\n

<\/canvas><\/p>\n

<\/p>\n<\/figure>\n

Two stage prediction with specialized classifiers.<\/p>\n

<\/p>\n

3. Low quality data<\/h4>\n

As said in the introduction, low quality <\/em>data<\/strong> will only lead to low quality <\/em>results<\/strong>.<\/p>\n

You might have samples in your dataset f your dataset that are too far from what you want to use. These might be more confusing<\/strong> for the model than helpful.<\/p>\n

Solution<\/em><\/strong>: remove<\/em> the worst images. This is a lengthy process, but will improve your results.<\/p>\n

\n

<\/canvas><\/p>\n

<\/p>

 <\/figcaption><\/figure>\n

Sure, these three images represent cats, but the model might not be able to work with it.<\/p>\n

Another common<\/strong> issue is when your dataset is made of data that doesn’t<\/strong>match<\/strong> the real<\/strong> world<\/strong> application<\/strong>. For instance <\/em>if the images are taken from completely different sources.<\/p>\n

Solution: <\/em><\/strong>think about the long term application of your technology, and which means will be used to acquire data in production. If possible, try to find\/build a dataset with the same tools.<\/p>\n

\n

<\/canvas><\/p>\n

<\/p>

Using data that doesn’t represent your real world application is usually a bad idea. Your model is likely to extract features that won’t work in the real world.<\/figcaption><\/figure>\n
\n

<\/canvas><\/p>\n

<\/p>\n<\/figure>\n

4. Unbalanced classes<\/h4>\n

If the number<\/strong> of sample per class isn’t<\/strong> roughly<\/em> the same<\/strong> for all classes, the model might have a tendency to favor the dominant class, as it results in a lower<\/strong> error<\/strong>. We say that the model is biased<\/strong> because the class distribution<\/em> is skewed<\/strong>. This is a serious issue, and also why you need to take a look at precision, recall<\/a> or confusion matrixes.<\/a><\/p>\n

Solution #1: <\/em><\/strong>gather more samples of the underrepresented <\/em>classes. However, this is often<\/strong> costly<\/strong> in time<\/em> and money<\/em>, or simply not feasible.<\/em><\/p>\n

Solution #2:<\/em><\/strong> over\/under-sample your data. This means that you remove<\/strong>some samples from the over-represented<\/strong> classes, and\/or duplicate<\/strong> samples from the under-represented<\/strong> classes. Better than duplication<\/em>, use data augmentation as seen previously.<\/p>\n

\n

<\/canvas><\/p>\n

<\/p>

We need to augment<\/strong> the under-represented <\/strong>class (cat) and leave aside <\/strong>some <\/strong>samples from the over-represented <\/strong>class (lime). This will give a much smoother class distribution.<\/figcaption><\/figure>\n
\n

<\/canvas><\/p>\n

<\/p>\n<\/figure>\n

5. Unbalanced data<\/h4>\n

If your data doesn’t have a specific<\/strong> format<\/strong>, or if the values don’t lie in the certain<\/strong> range<\/strong>, your model might have trouble dealing with it. You will have better results with image that are in aspect<\/em> ratio<\/em> and pixel<\/em> values<\/em>.<\/p>\n

Solution #1: <\/em><\/strong>Crop<\/em> or stretch<\/em> the data so that it has the same aspect or format<\/strong>as the other samples.<\/p>\n

\n

<\/canvas><\/p>\n

<\/p>

 <\/figcaption><\/figure>\n

Two possibilities to improve a badly formatted image.<\/p>\n

Solution #2: <\/em><\/strong>normalize<\/em> the data so that every sample has its data in the same<\/strong>value range<\/strong>.<\/p>\n

\n

<\/canvas> <\/p>

The value range is normalized to be consistent across the dataset.<\/figcaption><\/figure>\n
\n

<\/canvas><\/p>\n

<\/p>\n<\/figure>\n

6. No validation or testing<\/h4>\n

Once your dataset has been cleaned<\/em>, augmented<\/em> and properly labelled<\/em>, you need to split<\/strong> it. Many people split it the following way: 80%<\/em> for training<\/strong>, and 20% <\/em>for testing, <\/strong>which <\/strong>allow you to easily spot overfitting. <\/em>However<\/strong>, if you are trying multiple models on the same testing set, something else happens. By picking<\/em> the model giving the best test accuracy, you are in fact overfitting the testing set<\/strong>. This happens because you are manually selecting a model not<\/strong>for its intrinsic<\/strong> value<\/strong>, but for its performance<\/em> on a specific <\/strong>set of data.<\/p>\n

Solution:<\/em><\/strong> split the dataset in three: training<\/strong>, validation<\/strong> and testing<\/strong>. This shields<\/strong> your testing set from being overfitted<\/em> by the choice of the model<\/strong>. The selection process becomes:<\/p>\n

    \n
  1. Train<\/strong> your models on the training set.<\/strong><\/li>\n
  2. Test<\/strong> them on the validation set <\/strong>to make sure you aren’t overfitting<\/em>.<\/li>\n
  3. Pick the most promising model. Test<\/strong> it on the testing set<\/strong>, this will give you the true accuracy <\/strong>of your model.<\/li>\n<\/ol>\n
    \n

    <\/canvas><\/p>\n

    <\/p>\n<\/figure>\n

    Note<\/em><\/strong>: Once you have chosen<\/em> your model for production<\/strong>, don’t forget to train it on the whole<\/strong> dataset<\/strong>! The more data the better!<\/p>\n

    Conclusion<\/h4>\n

    I hope by now you are convinced<\/em> that you must pay attention to your dataset<\/strong>before even thinking about your model. You now know the biggest mistakes of working with data, how to avoid the pitfalls<\/strong>, plus tips<\/strong> and tricks<\/strong> on how to build killer<\/strong> datasets<\/strong>! In case of doubt, remember:<\/p>\n

    “The winner is not the one with the best model, it’s the one with the best data.”<\/strong>.<\/p><\/blockquote>\n<\/section>\n","protected":false},"excerpt":{"rendered":"

    Many people make the mistake of trying to compensate for their ugly dataset by improving their model. This is the equivalent of buying a super car because your old car doesn’t perform well with cheap gasoline. It makes much more sense to refine the oil instead of upgrading the car. This article explains how you can easily improve your results by enhancing your dataset, with the task of image classification as an example, but these tips can be applied to all sorts of datasets.<\/p>\n","protected":false},"author":389,"featured_media":3448,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[94],"ppma_author":[2196],"class_list":["post-976","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-data-science"],"authors":[{"term_id":2196,"user_id":389,"is_guest":0,"slug":"julien-despois","display_name":"Julien Despois","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2021\/05\/Screen-Shot-2019-10-07-at-5.45.59-PM-1-150x150.png","user_url":"","last_name":"Despois","first_name":"Julien","job_title":"","description":"Julien Despois is Machine Learning & Deep Learning Scientist at L\u2019Or\u00e9al AI Research. With a\u00a0strong background in Machine Learning, computer science, and mathematics he has a passion for Artificial Intelligence. He is specialized in applications of Deep Learning."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/976","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/389"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=976"}],"version-history":[{"count":1,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/976\/revisions"}],"predecessor-version":[{"id":6245,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/976\/revisions\/6245"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/3448"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=976"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=976"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=976"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=976"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}