Financial Machina: Machine Learning For Finance: The Quintessential Compendium for Python Machine Learning For 2024 & Beyond by Sampson Josh & Strauss Johann & Bisette Vincent & Van Der Post Hayden

Author:Sampson, Josh & Strauss, Johann & Bisette, Vincent & Van Der Post, Hayden
Language: eng
Format: epub
Publisher: Reactive Publishing
Published: 2024-01-06T00:00:00+00:00

4.5 Evaluation and Validation of Forecasting Models

The litmus test for any forecasting model is its ability to perform well on unseen data. This is where the concepts of overfitting and underfitting reveal their true impact. Overfitting occurs when a model learns not only the underlying pattern but also the noise within the training dataset, leading to a false sense of accuracy that falters on new data. Underfitting, conversely, is when a model fails to capture the underlying pattern altogether, producing poor predictions both on known and new data.

To navigate this delicate balance, several evaluation techniques are employed:

1. Out-of-Sample Testing: This involves setting aside a portion of the dataset, not seen by the model during the training phase, to test its predictions. This practice helps in assessing how the model will generalize to new, unseen data.

2. Cross-Validation: In this approach, the dataset is divided into 'k' subsets, or folds. The model is trained on 'k-1' folds and tested on the remaining one. This process is repeated 'k' times, with each fold serving as the test set once. The results are then averaged to produce a more robust performance estimate.

3. Walk-Forward Analysis: Particularly useful in time series forecasting, this method involves moving the training window forward in time and predicting the subsequent period. This is more aligned with real-world scenarios where models must predict future events based on past data.

4. Backtesting: A simulation where the model's predictions are compared against actual historical data. It is an excellent way to see how the model would have performed if it had been used in the past.

To complement these methods, a range of metrics is employed to quantify model performance. These include:

- Mean Absolute Error (MAE): This measures the average magnitude of the errors without considering their direction.

- Root Mean Squared Error (RMSE): This metric squares the errors before averaging, thus penalizing larger errors more heavily than the MAE.

- Mean Absolute Percentage Error (MAPE): This expresses the error as a percentage of the actual values, providing a sense of scale to the errors.

In Python, these metrics can be easily calculated using libraries such as `sklearn`:

```python

from sklearn.metrics import mean_absolute_error, mean_squared_error

# Assume 'actuals' is a numpy array of actual values and 'predictions' is the model's predictions

mae = mean_absolute_error(actuals, predictions)

rmse = mean_squared_error(actuals, predictions, squared=False)

```

While these metrics provide a quantitative measure of model performance, it is also important to consider the business context of the model's application. A model with a lower MAE might still be less valuable than one with a higher MAE if it fails to capture critical business events such as market crashes or surges.

Lastly, models must undergo robustness checks against different market conditions, stress tests to evaluate performance under extreme scenarios, and checks for potential biases that could skew the predictions. It is through these comprehensive evaluation and validation processes that forecasting models can be trusted to inform financial decisions and strategies, ensuring that their deployment is not only scientifically sound but also aligned with the practical realities of the financial markets.

4.5.1 Metrics and Criteria

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.