9.1 - Log-transforming Only the Predictor for SLR9.1 - Log-transforming Only the Predictor for SLR9.2 - Log-transforming Only the Response for SLR9.2 - Log-transforming Only the Response for SLR9.3 - Log-transforming Both the Predictor and Response9.3 - Log-transforming Both the Predictor and Response9.4 - Other Data Transformations9.4 - Other Data Transformations9.5 - More on Transformations9.5 - More on Transformations9.6 - Interactions Between Quantitative Predictors9.6 - Interactions Between Quantitative Predictors9.7 - Polynomial Regression9.7 - Polynomial Regression9.8 - Polynomial Regression Examples9.8 - Polynomial Regression ExamplesSoftware Help 9Software Help 9Minitab Help 9: Data TransformationsMinitab Help 9: Data TransformationsR Help 9: Data TransformationsR Help 9: Data Transformations↩ Show
Machine learning is a very iterative process. If performed and interpreted correctly, we can have great confidence in our outcomes. If not, the results will be useless. Approaching machine learning correctly means approaching it strategically by spending our data wisely on learning and validation procedures, properly pre-processing variables, minimizing data leakage, tuning hyperparameters, and assessing model performance. Before introducing specific algorithms, this tutorial introduces concepts that are commonly required in the supervised machine learning process and that you’ll see briskly covered in tutorials that follow. This tutorial will prepare you with the fundamentals needed prior to applying supervised machine learning algorithms.tl;drBefore introducing specific algorithms, this tutorial introduces concepts that you’ll see briskly covered in each chapter and are necessary for any type of supervised machine learning model:
PrerequisitesThis tutorial leverages the following packages.
To illustrate some of the concepts we will use the Ames Housing data that has been included in the
Data splittingSpending our data wiselyA major goal of the machine learning process is to find an algorithm that most accurately predicts future values () based on a set of inputs (). In other words, we want an algorithm that not only fits well to our past data, but more importantly, one that predicts a future outcome accurately. This is called the generalizability of our algorithm. How we “spend” our data will help us understand how well our algorithm generalizes to unseen data. To provide an accurate understanding of the generalizability of our final optimal model, we split our data into training and test data sets:
Given a fixed amount of data, typical recommendations for splitting your data into training-testing splits include 60% (training) - 40% (testing), 70%-30%, or 80%-20%. Generally speaking, these are appropriate guidelines to follow; however, it is good to keep in mind that as your overall data set gets smaller,
Typically, we are not lacking in the size of our data here, so a 70-30 split is often sufficient. The two most common ways of splitting data include simple random sampling and stratified sampling. Simple random samplingThe simplest way to split the data into training and test sets is to take a simple random sample. This does not control for any data attributes, such as the percentage of data in the quantiles in your response variable (). There are multiple ways to split our data. Here we show four options to produce a 70-30 split (note that setting the seed value allows you to reproduce your randomized splits):
Since this sampling approach will randomly sample across the distribution of ( Stratified samplingHowever, if we want to explicitly control our sampling so that our training and test sets have similar distributions, we can use stratified sampling. This is more common with classification problems where the reponse variable may be imbalanced (90% of observations with response “Yes” and 10% with response “No”). However, we can also apply to regression problems for data sets that have a small sample size and where the response variable deviates strongly from normality. With a continuous response variable, stratified sampling will break down into quantiles and randomly sample from each quantile. Consequently, this will help ensure a balanced representation of the response distribution in both the training and test sets. The easiest way to perform stratified sampling on a response variable is to use the
Feature engineeringFeature engineering generally refers to the process of adding, deleting, and transforming the variables to be applied to your machine learning algorithms. Feature engineering is a significant process and requires you to spend substantial time understanding your data…or as Leo Breiman said “live with your data before you plunge into modeling.” Although this guide is primarily concerned with machine learning algorithms, feature engineering can make or break an algorithm’s predictive ability. We will not cover all the potential ways of implementing feature engineering; however, we will cover a few fundamental pre-processing tasks that can significantly improve modeling performance. One-hot encodingMany models require all variables to be numeric. Consequently, we need to transform any categorical variables into numeric representations so that these algorithms can compute. Some packages automate this process
(i.e. The most common is referred to as one-hot encoding, where we transpose our categorical variables so that each level of the feature is represented as a boolean value. For example, one-hot encoding variable
results in the following representation:
This is called less than full rank (aka one-to-one) one-hot encoding where we retain all variables for each level of
If you
needed to manually implement one-hot encoding yourself you can with
Two things to note:
Response TransformationAlthough not a requirement, normalizing the distribution of the response variable by using a transformation can lead to a big improvement, especially
for parametric models. As we saw in the data splitting section, our response variable Fig 3: Right skewed response variable.To normalize, we have two options: Option 1: normalize with a log transformation. This will transform most right skewed distributions to be approximately normal.
Option 2: use a Box Cox transformation. A Box Cox transformation is more flexible and will find the transformation from a family of power transforms that will transform the variabe as close as possible to a normal distribution. Important note: be sure to compute the
We can see that in this example, the log transformation and Box Cox transformation both do about equally well in transforming our reponse variable to be normally distributed. Fig 4: Response variable transformations.Note that when you model with a transformed response variable, your predictions will also be in the transformed value. You will likely want to re-transform your predicted values back to their normal state so that decision-makers can interpret the results. The following code can do this for you:
Predictor TransformationSome models (K-NN, SVMs, PLS, neural networks) require that the features have the same units. Centering and scaling can be used for this purpose and is often referred to as standardizing the features. Standardizing numeric variables results in zero mean and unit variance, which provides a common comparable unit of measure across all the variables. Some packages have built-in arguments (i.e. For example, here we center and scale our predictor variables. Note, it is important that you standardize the test data based on the training mean and variance values of each feature. This minimizes data leakage.
Alternative Feature TransformationThere are some alternative transformations that you can perform:
Basic model formulationThere are many packages to perform machine learning and there are almost always more than one to perform each algorithm (i.e. there are over 20 packages to perform random forests). There are pros and cons to each package; some may be more computationally efficient while others may have more hyperparameter tuning options. Future tutorials will expose you to several packages; some that have become “the standard” and others that are new and may be considered “maturing”. Just realize there are more ways than one to skin a 🙀. For example, these three functions will all produce the same linear regression model output.
One thing you will notice throughout future tutorials is that we can specify
our model formulation in different ways. In the above examples we use the model formulation ( Matrix formulation requires that we separate our response variable from our features. For example, in the regularization tutorial we’ll use
Alternatively,
Model tuningHyperparameters control the level of model complexity. Some algorithms have many tuning parameters while others have only one or two. Tuning can be a good thing as it allows us to transform our model to better align with patterns within our data. For example, the simple illustration below shows how the more flexible model aligns more closely to the data than the fixed linear model. Fig 5: Tuning allows for more flexible patterns to be fit.However, highly tunable models can also be dangerous because they allow us to overfit our model to the training data, which will not generalize well to future unseen data. Fig 6: Highly tunable models can overfit if we are not careful.Throughout future tutorials we will demonstrate how to tune the different parameters for each model. However, we bring up this point because it feeds into the next section nicely. Cross Validation for GeneralizationOur goal is to not only find a model that performs well on training data but to find one that performs well on future unseen data. So although we can tune our model to reduce some error metric to near zero on our training data, this may not generalize well to future unseen data. Consequently, our goal is to find a model and its hyperparameters that will minimize error on held-out data. Let’s go back to this image… Fig 7: Bias versus variance.The model on the left is considered rigid and consistent. If we provided it a new training sample with slightly different values, the model would not change much, if at all. Although it is consistent, the model does not accurately capture the underlying relationship. This is considered a model with high bias. The model on the right is far more inconsistent. Even with small changes to our training sample, this model would likely change significantly. This is considered a model with high variance. The model in the middle balances the two and, likely, will minimize the error on future unseen data compared to the high bias and high variance models. This is our goal. Fig 8: Bias-variance tradeoff.To find the model that balances the bias-variance tradeoff, we search for a model that minimizes a k-fold cross-validation error metric (you will also be introduced to what’s called an out of bag error which provides a similar form of evaluation). k-fold cross-validation is a resampling method that randomly divides the training data into k groups (aka folds) of approximately equal size. The model is fit on folds and then the held-out validation fold is used to compute the error. This procedure is repeated k times; each time, a different group of observations is treated as the validation set. This process results in k estimates of the test error (). Thus, the k-fold CV estimate is computed by averaging these values, which provides us with an approximation of the error to expect on unseen data. Fig 9: Illustration of the k-fold cross validation process.Most
algorithms and packages we cover in future tutorials have built-in cross validation capabilities. One typically uses a 5 or 10 fold CV ( or ). For example,
Model evaluationThis leads us to our final topic, error metrics to evaluate performance. There are several metrics we can choose from to assess the error of a supervised machine learning model. The most common include: Regression models
Most models we assess in future tutorials will report most, if not all, of these metrics. We will often emphasize MSE and RMSE but its good to realize that certain situations warrant emphasis on some more than others. Classification models
When applying classification models, we often use a confusion matrix to evaluate certain performance measures. A confusion matrix is simply a matrix that compares actual categorical levels (or events) to the predicted categorical levels. When we predict the right level, we refer to this as a true positive. However, if we predict a level or event that did not happen this is called a false positive (i.e. we predicted a customer would redeem a coupon and they did not). Alternatively, when we do not predict a level or event and it does happen that this is called a false negative (i.e. a customer that we did not predict to redeem a coupon does). Fig 10: Confusion matrix.We can extract different levels of performance from these measures. For example, given the classification matrix below we can assess the following:
|