This course will help you slingshot the predictive capabilities of your models, far out-pacing the limits of out-of-box ML. From a ground-up perspective, we'll understand how a model functions, the part of the model that is able to fit the data on its' own, and how important additional tuning and fitting by a trained ML engineer is.
This module includes real-world examples, coding assignments, and lots of in-depth exploration of how and why model tuning should be done. If you understand the material in this course, your models will improve, and the results you will be able to deliver will as well!
What am I going to get from this course?
Exceed the performance of out-of-box machine learning models - drastically!
You will learn how to make sure you are getting the best predictions your model can
provide, if you should change your model, or even if you could benefit from
packaging multiple models together to get the best fit for your data.
This course is a crucial step towards becoming an expert data scientist - moving from
simply knowing how machine learning works, to understanding on an intuitive
level how these algorithms model data, and the state of the art techniques that
are available to improve their performance.
Prerequisites and Target Audience
What will students need to know or do before starting this course?
Students should have taken a basic course in machine learning, either in Experfy, or
have studied independently enough to understand the prerequisites listed above.
Additionally, they should have access to and ability with a basic programming
environment, such as python or R.
Who should take this course? Who should not?
This course is appropriate for anyone who will be involved in the planning and
implementation of machine learning or AI algorithms. Knowing how to tune a
model is a crucial step in using machine learning, so this course should be
taken by anyone who plans on using, maintaining, or understanding these models.
This course is hands-on and technical, and so would be of limited use to those who don't
plan on understanding the details of how a machine learning system is
Module 1: Model Tuning Introduction
Introduction and expectation-setting
Students will learn about model tuning - why it is important, and how it separates real machine learning engineers from beginners, or those just implementing out-of-box libraries.
This lecture will introduce the first topic in model tuning - hyper-parameters. We will go in detail about what they are, how they're used, and how understanding them can result in early, easy wins for the model.
Intro to Bayesianism
Bayesian reasoning is a powerful tool, especially when it comes to tuning machine learning models in complicated, incomplete environments. In this module we will review bayesian statistics, and discuss how it can be applied to ML models.
Intro to Bayesian Model Averaging
Bayesian model averaging is a powerful tool that can be used to drastically increase the performance of ML models. We'll give a theoretical framework for it in this module.
Bayesian Model Averaging- Specification
Continuing our delve into Bayesian Model Averaging, we'll go over a practical example, and show how it might be applied to problems you encounter.
Discussing the theoretical difficulties surrounding BMA, we'll show some tricks used to overcome these difficulties, including Occam's Window.
Computing the Integral
This module finishing up our discussion of the theoretical difficulties with BMA, discussing the difficulties in computing complex integrals, and reviewing some techniques used to either compute or approximate it.
Bayesian Model Averaging-Worked Example
Finally, we'll delve into a real example of BMA and show how it can deliver great results - and set you up to try one yourself!
Intro to Bootstrap Aggregation
Bootstrap Aggregation is a key concept in the construction of more sophisticated and powerful machine learning models. We'll discuss it's concept and implementation here.
Intro to Bootstrap Aggregation- CART
We'll work an example of bagging using it's application to classification and regression trees, or CARTs. This is a good intro to how bagging is used in the real world.
Problem with Bagged Decision Trees
Why not be content with bagged decision trees? We'll discuss problems with that framework, and suggest a simple way they can be fixed with the intro to boosting.
Random Forests- Start to Finish
Random Forests by themselves can be powerful ML tools, so let's first understand how they work, and then show how boosting can make them truly powerful!
Random Forests: Time-Accuracy Tradeoff
One concept to always keep in mine when tuning a model is the time-accuracy tradeoff. At this point in the lecture series, we know enough to illustrate how if we used every tool exhaustively, this can be a prohibitive time investment. Let's review it here!
Boosted Trees- Differences from Random Forest
Building on the framework we have for understanding random forests, let's now delve deeply into how this basic setup can be used to accommodate boosting!
Boosted trees- Adaboost Procedure
Adaboost is very close to being best-in-class, and it's a simple implementation of boosted trees. This module clearly explains the whats and hows of Adaboost.
XG Boost- Gradient Boosting
The final piece of the puzzle needed to get us to the best-in-class XGBoost algorithm is gradient boosting. We'll explore that concept here, and start to see how it can apply to our algorithm.
Boosted Trees- Final Decision
Apply everything we've learned, let's see how XGBoost and boosted trees can work on real-world problems.
Module 4: Hyper-Parameters
Introduction to hyper-parameters- Basics
To truly understand a model, you must know which parts of the algorithm are tuned by the model itself, and which you must adjust. Let's review hyper-parameters, and how important they are to model tuning.
Hyperparameters in decision trees
Decision trees are an easily-understandable ML model (and the basis of much of what we've discussed!). Let's look at hyper-parameters in the context of these models.
Hyperparamters in decision trees- Levels
Understanding tradeoffs is a major part of model tuning. Explore how adjusting a hyper-parameter gives us very different results in a decision tree.
Hyperparameters in decision trees- AUC
With so many potential values for hyper-parameters, how do we know which to pick? Let's investigate AUC as a measure of model fitness.
Finding optimal hyperparameters- Brute Force
How to find the optimal hyper-parameter? Sometimes the best ideas to start with are the simplest - let's consider the pros and cons of the brute force method.
Finding optimal hyperparameters- Sanity Check
Understanding tradeoffs is a recurring theme. Let's understand the choices we're making, and think how to best find the happy medium that makes for good hyper-parameters.
Intro to Stacking
Model stacking is a competition-winner, suitable for problems that would otherwise be intractable. Let's lay down the foundation and understand the concepts behind it.
Intro to stacking
The best way to understand is to see the basic concepts in action - here we'll review those concepts, and begin to see their application.
Intro to stacking- Motivation
Why should model stacking be able to deliver such dramatic results? Some examples show precisely why this technique is so powerful.
Now that we understand the theory, let's see some of the real-world results model stacking has been able to accomplish.
Let's remember the basics of model tuning - why we need to tune, the first steps, and how it sets us apart from those who don't fully understand it.
Even the most complicated models we saw started with a comprehensible framework. Let's discuss the solid foundation required for effective model tuning.
Know your data
One of the most important mantras for any data scientist, "know your data" is a crucial step before tuning the data - models should fit the data!
We review the tradeoff that we've been discussing for this lecture series. How do we intelligently decide when the model is fit enough, and the rewards are diminishing?
Example scenario- Network transactions
Finally, let's end with another comprehensive example - this should tie together the course and illustrate our points!