Example 1: Classification via Boosted Trees

We'll use the default analysis options in the Boosted Trees for Regression and Classification module. Select Boosted Tree Classifiers and Regression from the Data Mining menu to display the Boosted Trees Startup Panel.

Because the goal of this analysis is to build a prediction model for the different digits based on several (faulty) categorical predictor variables, accept the default Classification Analysis option and click OK to display the Boosted Trees Specifications dialog.

On the Quick tab, click the Variables button to display the variable selection dialog, select Digit as the categorical dependent variable and all others as the categorical predictor variables, and click the OK button to return to the Specifications dialog.

There are a number of additional options available on the Classification and Advanced tabs of this dialog, which can be used to "fine-tune" the analysis. You can use the options on the Classification tab to specify particular a priori class probabilities and unequal misclassification costs; using the options on the Advanced tab, you can determine the complexity of the individual trees you want to build in each boosting step, as well as the total number of boosting steps.

For the purpose of this analysis (and as a useful "first step" for most analyses), let's accept the defaults; thus, click OK. You will see the Computing dialog for a few moments as the consecutive boosting steps are computed, and then the Results dialog will be displayed.

First, click the Summary button to review how consecutive boosting steps improved the accuracy (quality of the prediction model) for the randomly selected training data and testing data.

This graph demonstrates the basic mechanism of how the stochastic gradient boosting algorithm implemented in Statistica can avoid overfitting (see also the Introductory Overview). As more and more additive terms (simple trees) are added to the model, the average squared error function for the training data (from which the respective trees were estimated) will decrease. However, the error estimate for the training data will at one point start to increase, clearly marking the point where evidence for overfitting is beginning to show.

By default, the program will designate 54 as the optimal number of trees (in this case; because of the random subsampling of training data in successive boosting steps, your results may be slightly different); this happens to be the point where the smallest error for the testing data occurred. You can use the Number of trees option on the Boosted Trees Results dialog - Quick tab to select a specific solution, i.e., number of trees in the final model.

Next, click on the Classification tab of the Results dialog, select the Test set option button in the Sample group box, and then click the Predicted vs. observed by classes button. The program will then compute the matrix of predicted and observed classification frequencies, which are displayed in a results spreadsheet, as well as a bivariate histogram.

As you can see, the final solution is remarkably accurate over all digits. You may want to review the various additional summary statistics (e.g., Risk estimates) to gage the quality of different solutions, i.e., for different numbers of additive terms (simple trees).

Another way to assess the accuracy of the prediction is to compute the gains chart for each category of the dependent variable. Select the Classification tab of the Results dialog to compute gains or lift charts for different samples.

In general, stochastic gradient boosting of simple trees (see the Introductory Overview) is best considered a machine learning algorithm, i.e., a "black box" to some extent that will (usually) produce very accurate predictions, but yield models that are not easily interpretable (unlike, for example, classification and regression trees or linear models, where the final prediction model can usually be "put into words," i.e., explained). To interpret the results from the Statistica Boosted Trees module, there are two key tools:

The bar plot and spreadsheet of the predictor importance usually shed some light on the variables that make the major contributions to the prediction of the dependent variable of interest. For example, shown below is the bar plot of predictor importance for the current example.

The predictor importance is computed as the relative (scaled) average value of the predictor statistic over all trees. So, for example, in this case it is the average value of the sums-of-squares prediction over all categories and over all trees and nodes, scaled so that the maximum value of that sum is equal to 1. Hence, these values reflect on the strength of the relationship between the predictors and the dependent variable of interest over the successive boosting steps. In this case, variables Var2, Var4, and Var5 stand out as the most important predictors.

You can also review the final sequence of (boosted) trees, either graphically or in a sequence of results spreadsheets (one for each tree). However, this may not be a useful way to examine the "meaning" of the final model when the final solution involves a large number of additive terms (simple trees). In this case, the final solution involves 54 binary trees, each with a simple split. So, most likely, in this case it would not be very useful to examine those trees.

Finally, you can deploy the model via the Code generator on the Results dialog - Report tab. In particular, you may want to save the PMML deployment code for the created model, and then use that code via the Rapid Deployment Engine module to predict (classify) new cases.