Modeling

Machine Learning with R

R caret rpart randomForest class e1701 stats factoextra

By Afshine Amidi and Shervine Amidi

Overview

There are several steps that are needed to build a machine learning model:

• feature engineering: building features that can be interpreted and that can have a high predictive power
• model selection: choosing a model that can generalize well on unseen data
• tracking performance metrics: quantifying how the model performs and include the uncertainty of predictions in the results

Data preparation

Resampling The following table summarizes the main sampling techniques that are used to correct the distribution of the data, which can be necessary when classes are imbalanced:

Action on		Command	Illustration
minority class	majority class
		(x, y)
	Downsample	downSample(x, y)
Upsample with repetitions		upSample(x, y)
Upsample with SMOTE		SMOTE(form, data)

Relationship between variables The following command enables to display pairwise correlations as well as visual comparisons using the GGally library:

ggpairs(data)

The matrix plot generated looks like the following:

where the representation at row $i$ and column $j$:

• gives the correlation number between variables $i$ and $j$ when $i < j$
• plots the probability distribution of variable $i$ when $i = j$
• plots the data in 2D where $x$- and $y$-axes are resp. variables $j$ and $i$ when $i > j$

Remark: this command is useful for variable selection decisions.

Scaling When features of a dataset are on different scales, it is sometimes recommended to standardize them. This operation can be done using the caret library as follows:

# Create scaling based on reference dataset data_reference
scaling = preProcess(data_reference, method = c('center', 'scale'))

# Apply scaling to dataset data
data_scaled = predict(scaling, data)

Data splits The data can be randomly split into proportions $p$ and $1-p$ resp. for training and validation sets using the caret library.

One can do so with the following commands:

# Create partition
train_partition = createDataPartition(y, p, list = FALSE)

# Split data into train and test sets
data_train = data[train_partition,]
data_val = data[-train_partition,]

Supervised models

In this part, we want to learn the relationship between the data and its labels. It can either be a regression (continuous output) or classification (multi-class output).

General methods

Linear regression Linear regression is a supervised learning method used for regression problems.

Given a data frame data containing the independent variables x and the dependent variable y, along with a formula of the type y ~ x, we can build a linear regression model as follows:

model = lm(data, formula)

predictions = predict(object, newdata, params)

where the parameters are summarized in the table below:

Remark: check if predictors are multicollinear. Can be used to interpret association between predictors and output.

Logistic regression Logistic regression is a supervised learning method for classification problems.

Given a data frame data containing the independent variables x and the dependent variable y, along with a formula of the type y ~ x, we can build a logistic regression model as follows:

model = glm(data, formula, family = 'binomial')

k-NN The k-nearest neighbors algorithm, also known as k-NN, is a supervised method.

Given training features train along with labels cl as well as test features test, we can compute predictions using the class library with the following command:

predictions = knn(train, test, cl, params)

SVM Support vector machines (SVM) is algorithm that can be used both for regression (SVR) and classification (SVC).

The following command creates a model using the e1701 library:

model = svm(data, formula, params)

We can make predictions with:

predictions = predict(object, newdata, params)

where the parameters are summarized in the table below:

Decision tree Decision trees are one type of tree-based models that can be built for either classification or regression purposes.

The following command uses the rpart library as follows:

model = rpart(data, formula, params)

The resulting tree can be interpreted using commands coming from the rpart.plot library, that are summarized in the table below:

Ensemble models

Random forest Random forest is a tree-based ensemble model.

The following command uses the randomForest library:

model = randomForest(data, formula, params)

We can make predictions as follows:

predictions = predict(model, newdata, params)

where the parameters are summarized in the table below:

XGBoost Gradient boosting models can be trained using the xgboost library.

First the data needs to be in DMatrix format, which can be done with the following command:

data_xgb = xgb.DMatrix(data, label)

model = xgb.train(data, label, params)

Predictions are made as follows:

predict(object, newdata, params)

Model performance

Cross-validation Cross-validation is a technique used to find the optimal hyperparameters of a model by making sure that it does not overfit the training set. It divides the dataset into $k$ splits, and trains the model on $k-1$ splits while making predictions on the $k^{th}$ one.

This can be achieved with the caret library that will fit the resulting model on the training set with the cross-validated optimal hyperparmeters.

First, the training setting is defined as follows:

# Set partitions
train_control = trainControl(params)

where the parameters are summarized in the table below:

Then, a model can be fit using the control parameters as follows:

# Find optimal parameters with cross-validation
model = train(data, formula, trControl = train_control, method, metric, maximize, tuneGrid)

The parameters are summed up in the table below:

Regression metrics The table below summarizes the main metrics used in regression problems, based on predicted pred and actuals act, by using the caret library:

Classification metrics The table below summarizes the main metrics used in classification problems, that can be accessed with the caret, ModelMetrics and the PRROC libraries:

Unsupervised models

In this part, we do not have true labels, and we want to get insights from the data.

k-means The $k$-means algorithm can be performed using the base stats library.

By taking as input a matrix of features x as well as additional parameters, we have:

model = kmeans(x, params)

The attributes of the model can then be accessed with the dimensions summarized below:

The elbow method along with the distortion metric is used to select the number of clusters that make the most sense.

Remark: for reproducibility purposes, it is important to the seed to a fixed value with the set.seed() command.

Hierarchical clustering The hierarchical clustering algorithm is an unsupervised technique meant at forming clusters of observations that look alike based on their distances. It can be performed using the base stats library:

# Dissimilarity matrix
df_distance = dist(data, method = "euclidean")

# Hierarchical clustering using Complete Linkage
hc = hclust(d, method = "complete")

PCA Principal components analysis, commonly known as PCA, is a dimension reduction technique that can be used as a data preparation step to reduce the number of variables.

It is done using the prcomp function on a matrix x as follows:

model = prcomp(x, params)

where the parameters are summarized below:

A scree plot is commonly used to check how much of the variance is explained by the newly-created principal components. The table below summarizes the main actions that can be done using the factoextra library: