Applied Predictive Modeling by Max Kuhn & Kjell Johnson

Author:Max Kuhn & Kjell Johnson
Language: eng
Format: epub
Publisher: Springer New York, New York, NY

Observed class

Successful

Unsuccessful

Successful

490

220

Unsuccessful

80

767

This model had an overall accuracy of 80.7 %, a sensitivity of 86 %, and a specificity of 77.7 %. The reduced set of predictors were used to generate this matrix

Recall that PLSDA encodes the response as a set of 0/1 dummy variables. Since PLS is a linear model, the predictions for the PLSDA model are not constrained to lie between 0 and 1. The final class is determined by the class with the largest model prediction. However, the raw model predictions require post-processing if class probabilities are required. The softmax approach previously described in Sect.​ 11.​1 can be used for this purpose. However, our experience with this technique is that it does not produce meaningful class probabilities—the probabilities are not usually close to 0 or 1 for the most confident predictions. An alternative approach is to use Bayes’ Rule to convert the original model output into class probabilities (Fig. 12.13). This tends to yield more meaningful class probabilities. One advantage of using Bayes’ Rule is that the prior probability can be specified. This can be important when the data include one or more rare classes. In this situation, the training set may be artificially balanced and the specification of the prior probability can be used to generate more accurate probabilities. Figure 12.14 shows the class probabilities for the year 2008 grants.11 While there is overlap between the classes, the distributions are properly shifted; probabilities tend to be higher for the truly successful grants and low for the unsuccessful grants.

Fig. 12.13The ROC curve for the 2008 holdout data using PLS (black). The AUC is 0.89. The ROC curve for LDA and logistic regression are overlaid (grey) for comparison. All three methods perform similarly

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.