Learning Scikit-Learn by Machine Learning in Python

Author:Machine Learning in Python
Format: epub

Our tree has an accuracy of 0.838 on the training set. But remember that this is not a good indicator. This is especially true for decision trees as this method is highly susceptible to overfitting. Since we did not separate an evaluation set, we should apply cross-validation. For this example, we will use an extreme case of cross-validation, named leave-one-out cross-validation. For each instance in the training sample, we train on the rest of the sample, and evaluate the model built on the only instance left out. After performing as many classifications as training instances, we calculate the accuracy simply as the proportion of times our method correctly predicted the class of the left-out instance, and found it is a little lower (as we expected) than the resubstitution accuracy on the training set.

>>> from sklearn.cross_validation import cross_val_score, LeaveOneOut >>> from scipy.stats import sem >>> >>> def loo_cv(X_train, y_train,clf): >>> # Perform Leave-One-Out cross validation >>> # We are preforming 1313 classifications! >>> loo = LeaveOneOut(X_train[:].shape[0]) >>> scores = np.zeros(X_train[:].shape[0]) >>> for train_index, test_index in loo: >>> X_train_cv, X_test_cv = X_train[train_index], X_train[test_index] >>> y_train_cv, y_test_cv = y_train[train_index], y_train[test_index] >>> clf = clf.fit(X_train_cv,y_train_cv) >>> y_pred = clf.predict(X_test_cv) >>> scores[test_index] = metrics.accuracy_score( y_test_cv.astype(int), y_pred.astype(int)) >>> print ("Mean score: {0:.3f} (+/-{1:.3f})").format(np.mean(scores), sem(scores)) >>> loo_cv(X_train, y_train,clf) Mean score: 0.837 (+/-0.012)

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.