{"id":22516,"date":"2020-12-22T09:47:58","date_gmt":"2020-12-22T09:47:58","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/what-is-ensemble-learning\/"},"modified":"2023-09-14T17:39:47","modified_gmt":"2023-09-14T17:39:47","slug":"what-is-ensemble-learning","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/what-is-ensemble-learning\/","title":{"rendered":"What Is Ensemble Learning?"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The principle of \u201cthe wisdom of the crowd\u201d shows that a large group of people with average knowledge on a topic can provide reliable answers to questions such as predicting quantities, spatial reasoning, and general knowledge. The aggregate results cancel out the noise and can often be superior to those of highly knowledgeable experts. The same rule can apply to artificial intelligence applications that rely on\u00a0machine learning<\/a>, the branch of AI that predicts outcomes based on mathematical models.<\/p>\n\n

In machine learning, crowd wisdom is achieved through ensemble learning. For many problems, the result obtained from an ensemble, a combination of machine learning models, can be more accurate than any single member of the group.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

How does ensemble learning work?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Say you want to develop a machine learning model that predicts inventory stock orders for your company based on historical data you have gathered from previous years. You use train four machine learning models using a different algorithm: linear regression, support vector machine, a regression decision tree, and a basic\u00a0artificial neural network<\/a>. But even after much tweaking and configuration, none of them achieves your desired 95 percent prediction accuracy. These machine learning models are called \u201cweak learners\u201d because they fail to converge to the desired level.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"disparate\t\t\t\t\t\t\t\t\t\t\t
Single machine learning models do not provide the desired accuracy<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

But weak doesn\u2019t mean useless. You can combine them into an ensemble. For each new prediction, you run your input data through all four models, and then compute the average of the results. When examining the new result, you see that the aggregate results provide 96 percent accuracy, which is more than acceptable.<\/p>\n\n

The reason ensemble learning is efficient is that your machine learning models work differently. Each model might perform well on some data and less accurately on others. When you combine all them, they cancel out each other\u2019s weaknesses.<\/p>\n\n

You can apply ensemble methods to both predictions problems, like the inventory prediction example we just saw, and classification problems, such as determining whether a picture contains a certain object.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"ensemble\t\t\t\t\t\t\t\t\t\t\t
Ensemble machine learning combine several models to improve the overall results.<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Ensemble methods<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

For a machine learning ensemble, you must make sure your models are independent of each other (or as independent of each other as possible). One way to do this is to create your ensemble from different algorithms, as in the above example.<\/p>\n\n

Another ensemble method is to use instances of the same machine learning algorithms<\/a> and train them on different data sets. For instance, you can create an ensemble composed of 12 linear regression models, each trained on a subset of your training data.<\/p>\n\n

There are two key methods for sampling data from your training set. \u201cBootstrap aggregation,\u201d aka \u201cbagging,\u201d takes random samples from the training set \u201cwith replacement.\u201d The other method, \u201cpasting,\u201d draws samples \u201cwithout replacement.\u201d<\/p>\n\n

To understand the difference between the sampling methods, here\u2019s an example. Say you have a training set with 10,000 samples and you want to train each machine learning model in your ensemble with 9,000 samples. In case you\u2019re using bagging, for each of your machine learning models, you take the following steps:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
  1. Draw a random sample from the training set.<\/li>
  2. Add a copy of the sample to the model\u2019s training set<\/li>
  3. Return the sample to the original training set<\/li>
  4. Repeat the process 8,999 times<\/li><\/ol>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\"bagging\t\t\t\t\t\t\t\t\t\t\t
    Bagging sampling draws samples from the training set and replaces them<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    When using pasting, you go through the same process, with the difference that samples are not returned to the training set after being drawn. Consequently, the same sample might appear in a model\u2019s several times when using bagging but only once when using pasting.<\/p>\n\n

    After training all your machine learning models, you\u2019ll have to choose an aggregation method. If you\u2019re tackling a classification problem, the usual aggregation method is \u201cstatistical mode,\u201d or the class that is predicted more than others. In regression problems, ensembles usually use the average of the predictions made by the models.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\"pasting\t\t\t\t\t\t\t\t\t\t\t
    Pasting draws samples from the training set and replaces them<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    Boosting methods<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Another popular ensemble technique is \u201cboosting.\u201d In contrast to classic ensemble methods, where machine learning models are trained in parallel, boosting methods train them sequentially, with each new model building up on the previous one and solving its inefficiencies.<\/p>\n\n

    AdaBoost (short for \u201cadaptive boosting\u201d), one of the more popular boosting methods, improves the accuracy of ensemble models by adapting new models to the mistakes of previous ones. After training your first machine learning model, you single out the training examples misclassified or wrongly predicted by the model. When training the next model, you put more emphasis on these examples. This results in a machine learning model that performs better where the previous one failed. The process repeats itself for as many models you want to add to the ensemble. The final ensemble contains several machine learning models of different accuracies, which together can provide better accuracy. In boosted ensembles, the output of each model is given a weight that is proportionate to its accuracy.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t
    \n\t\t\t
    <\/div>\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    Random forests<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    One area where ensemble learning is very popular is decision trees, a machine learning algorithm that is very useful because of its flexibility and interpretability. Decision trees can make predictions on complex problems, and they can also trace back their outputs to a series of very clear steps.<\/p>\n\n

    The problem with decision trees is that they don\u2019t create smooth boundaries between different classes unless you break them down into too many branches, in which case they become prone to \u201coverfitting,\u201d a problem that occurs when a machine learning model performs very well on training data but poorly on novel examples from the real world.<\/p>\n\n

    This is a problem that can be solved through ensemble learning. Random forests are machine learning ensembles composed of multiple decision trees (hence the name \u201cforest\u201d). Using random forests ensures that a machine learning model does not get caught up in the specific confines of a single decision tree.<\/p>\n\n

    Random forests have their own independent implementation in Python machine learning libraries such as scikit-learn.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t

    Challenges of ensemble learning<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"random\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tp>While ensemble learning is a very powerful tool, it also has some tradeoffs.<\/p>\n\n

    Using ensemble means you must spend more time and resources on training your machine learning models. For instance, a random forest with 500 trees provides much better results than a single decision tree, but it also takes much more time to train. Running ensemble models can also become problematic if the algorithms you use require a lot of memory.<\/p>\n\n

    Another problem with ensemble learning is\u00a0explainability<\/a>. While adding new models to an ensemble can improve its overall accuracy, it makes it harder to investigate the decisions made by the AI algorithm. A single machine learning models such as decision tree is easy to trace, but when you have hundreds of models contributing to an output, it is much more difficult to make sense of the logic behind each decision.<\/p>\n\n

    As with most everything you\u2019ll encounter in machine learning, ensemble is one among the many tools you have for solving complicated problems. It can get you out of difficult situations, but it\u2019s not a silver bullet. Use it wisely.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

    In machine learning, crowd wisdom is achieved through ensemble learning. For many problems, the result obtained from an ensemble, a combination of machine learning models, can be more accurate than any single member of the group.<\/p>\n","protected":false},"author":109,"featured_media":16987,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[97,1157,92],"ppma_author":[1946],"class_list":["post-22516","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-artificial-intelligence","tag-ensemble-learning","tag-machine-learning"],"authors":[{"term_id":1946,"user_id":109,"is_guest":0,"slug":"ben-dickson","display_name":"Ben Dickson","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2020\/04\/medium_8aaf6bea-c4c1-455f-8156-8007d70910f8-150x150.jpg","user_url":"https:\/\/bdtechtalks.com\/","last_name":"Dickson","first_name":"Ben","job_title":"","description":"Ben Dickson is an experienced software engineer and tech blogger. He contributes regularly to major tech websites such as the Next Web, the Daily Dot, PCMag.com, Cointelegraph, VentureBeat, International Business Times UK, and The Huffington Post."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22516","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/109"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=22516"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22516\/revisions"}],"predecessor-version":[{"id":32984,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22516\/revisions\/32984"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/16987"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=22516"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=22516"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=22516"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=22516"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}