Ensemble Tree Methods

Ensemble Methods

Back to Homepage

Ensemble decision tree methods improve upon simple decision trees by averaging the outcomes of many decision trees. By taking the average of many models, we reduce variance and typically improve the prediction error rate.

install.packages(“MASS”)

install.packages(“randomForest”)

library(MASS)

library(randomForest)

The data set is Housing Values in Suburbs of Boston. There are 506 observations and 14 features.

data(Boston)
?Boston

Random Forest

Create a training data set using 300 of the 506 observations.

set.seed(101)
train <- sample(1:nrow(Boston),300)

Use the randomForest function to generate the random forest model using the training set. The target variable is medv, the median home value of owner-occupied homes.

rf.Boston = randomForest(medv~., data=Boston, subset=train)
rf.Boston

Call:
 randomForest(formula = medv ~ ., data = Boston, subset = train) 
               Type of random forest: regression
                     Number of trees: 500
No. of variables tried at each split: 4

          Mean of squared residuals: 12.30718
                    % Var explained: 85.13

The first model used all available predictors to split the nodes of the decision trees. We can adjust this value to see if there is a different.

Create two vector variables to hold out-of-bag error and test error for each model.

oob.err <- double(13)
test.err <- double(13)

mtry is a parameter of the randomForest function corresponding to the number of features to randomly select. We create a loop that will fit models using 1, 2, 3 randomly selected features for each split, up to 13 which is all the features.

Four hundred trees will be created for each value of mtry.

For each value of mtry the average Mean Squared Error (MSE) will be captured in the oob.err variable.

We then predict the values of medv for the observations not in the training set.

For each value of mtry the Residual Squared Error (RSS) will be captured in the test.err variable.

Finally, the values of mtry are printed out just to show the loop functioned.

for (mtry in 1:13) {
  fit <- randomForest(medv~., data=Boston, subset=train, mtry = mtry, ntree=400)
  oob.err[mtry] <- fit$mse[400]
  pred <- predict(fit, Boston[-train,])
  test.err[mtry] <- with(Boston[-train,], mean((medv-pred)^2))
  cat(mtry," ")
}

1  2  3  4  5  6  7  8  9  10  11  12  13  

We plot the MSE and RSS for each value of mtry using the matplot function.

matplot(1:mtry, cbind(test.err,oob.err), pch=19, col=c("red","blue"), type="b", ylab="Mean Squared Error")
legend("topright", legend=c("OOB", "Test"), pch=19, col=c("red","blue"))

The mean squared error of the predictions drops from 26 (when only a single feature is used) to a minimum around 12 when five random features are used to split the data at each node.

Boosting an Ensemble Decision Tree Method

Boosting is very similar to random forest, except that each new tree seeks to improve upon the performance of the previous tree. The trees are created sequentially and are shallow, possibly limited to two or four terminal nodes. Boosting reduces bias in a model helping it perform better on test data.

Boosting may be performed with the gbm package, Gradient Boosted Machines.

require(gbm)

Loading required package: gbm
package <U+393C><U+3E31>gbm<U+393C><U+3E32> was built under R version 3.3.3Loading required package: survival
Loading required package: lattice
Loading required package: splines
Loading required package: parallel
Loaded gbm 2.1.3

There are three tuning parameters for boosting.

1. The number of trees
2. The shrinkage parameter lambda
3. The number of splits allowed for each tree

Call gbm to create 10,000 trees. We pick an arbitrary value for lambda - 0.01, labeled shrinkage. Interaction depth indicates the number of splits. We limit each tree to four splits. The distribution is set based on the loss function of the algorithm. Ours is Gaussian because we are measuring success based on squared error.

boost.boston <- gbm(medv~., data = Boston[train,], distribution = "gaussian", n.trees=10000, shrinkage = 0.01, interaction.depth=4)
summary(boost.boston)

The summary output is a variable importance plot. The number of rooms and the percentage of “low status” people appear to be the most important predictors of medv.

We can plot the relationships between medv and each variable from the boosting object.

plot(boost.boston, i="rm")

We can see that as the number of rooms increases, medv (as a function of rm) increases, especially after seven rooms.

Finally, we will predict values from the test data set using the boosted model.

• We create a variable to hold the number 100 through 10,000, incrementally increased by 100.

• We create a variable to hold our preduction on the test data set using the boost.boston boosted decision tree model.

• The dim function shows that the output of the prediction contains 100 observations corresponding to the number of models we tried with larger numbers of trees from n.trees.

• We create the variable berr to hold the residual sume of squares from the predictions.

• We plot the number of trees (we have 100 models each with a different number of trees) against the residual sum of squares (RSS).

• Finally we add a horizontal line to the plot comparing the RSS to the test error from the unboosted random forest model.

n.trees = seq(from=100, to=10000, by=100)
predmat = predict(boost.boston, newdata = Boston[-train,], n.trees=n.trees)
dim(predmat)

[1] 206 100

berr = with(Boston[-train,], apply( (predmat-medv)^2,2,mean) )
plot(n.trees, berr, pch=19, ylab="Mean Squared Error", xlab="# Trees", main="Boosting Test Error")
abline(h=min(test.err), col="green")

We see that the boosted model reduces the test error significantly but after ~3,000 trees, the effect is minimized the the error flattens out.