This is a comprehensive guide to backtesting with skforecast in python with examples. Backtesting in time series forecasting simulates how a model would perform if it had been applied in the past, based on historical data. This process is crucial for validating models before deployment, as it offers insights into potential forecasting errors and helps optimize model performance. By testing how models perform under different scenarios, forecasters can better understand limitations and avoid overfitting.
For instance, backtesting allows you to assess how a model might perform during different seasonal cycles or economic conditions, enabling a more robust evaluation. In this guide, we’ll leverage Skforecast, a Python library that integrates seamlessly with scikit-learn, to facilitate backtesting for time series models.
Backtesting is a validation technique in time series analysis that evaluates the accuracy of forecasting models by testing them on historical data. In forecasting web traffic, backtesting methods help refine predictions and improve model reliability by revealing how they would perform in real-world scenarios. This process is crucial for both web traffic and financial backtesting, as it provides insights into time series trends and supports robust prototyping.
Utilizing a variety of forecaster types and validation strategies, backtesting in time series forecasting often involves tools for training models with machine learning, enhancing predictions with a range of functionalities. Effective backtesting captures the intricacies of time series data, addressing essential aspects such as seasonal variations and feature requests for more specific trends. By modeling time series trends with these advanced techniques, analysts can ensure their models are both accurate and adaptable to changing patterns, making backtesting a core component of forecasting methodologies.
Backtesting is widely used across various fields, including intermittent demand forecasting, where it assesses models that predict irregular or sporadic demand patterns. For instance, forecasting energy demand requires handling exogenous variables like weather conditions or public holidays, which significantly impact usage. By incorporating series differentiation and using recursive multi-step forecasting, models can accommodate the complexities of demand with machine learning, improving accuracy. Forecaster objects and time series differentiation enable more precise modeling, while exogenous features capture additional influences that affect energy consumption.
In the context of PV (photovoltaic) power forecasting, backtesting aids in building interpretable forecasting models that can predict power demand and output. Multi-series forecasting is especially relevant here, as energy production data often involve multiple time series, such as different geographic regions or types of PV systems. Techniques like direct multi-step forecasting are used to generate probabilistic forecasts, which provide a range of possible outcomes. These forecasts, combined with quantile regression, offer robust estimates of PV output under various conditions, making the models more resilient to data variability.
For more advanced applications, such as forecasting energy demand with machine learning, a hyperparameter search via random search or k-fold cross-validation fine-tunes model performance. Tools like quantile forecasts and categorical features help produce probabilistic forecasts, giving a more comprehensive view of potential future states. These methods support both single and multi-step forecasters, allowing analysts to explore the implications of different forecast periods. With interpretable forecasting models and robust backtesting techniques, forecasters can better handle the intricacies of time series data and produce reliable forecasts that are crucial for effective decision-making.
What is Skforecast?
Skforecast simplifies time series forecasting by automating tasks like lag creation and offering a wide selection of models through scikit-learn. This tool is suitable for both beginners and advanced users interested in performing autoregressive forecasting and backtesting due to its intuitive interface and support for custom model integrations.
Skforecast’s primary functionality centers on the ForecasterAutoreg class, which is pivotal for implementing autoregressive forecasting models. This class is effective for applications like demand forecasting and stock price prediction by incorporating lag values. Lags allow models to incorporate past values into future forecasts, enhancing accuracy and predictive power. The development of Skforecast focuses on making forecasting accessible while enabling complex forecasting strategies such as recursive and direct approaches.
The recursive strategy, or series recursive, uses previous forecasted values as inputs for future predictions, which is ideal for capturing ongoing patterns but may propagate errors. The direct strategy, on the other hand, forecasts each time step individually, offering a realistic assessment for multi-step forecasts, reducing cumulative error. By combining Skforecast with gradient boosting models, users can achieve robust forecasting and obtain prediction intervals to gauge forecast certainty. This range of functionalities makes Skforecast an adaptable tool for numerous real-world forecasting tasks.
Key Features of Skforecast
Seamless Integration with scikit-learn: Skforecast enables the use of scikit-learn regressors, making it easy to try different models.
Support for Multiple Backtesting Techniques: Skforecast offers both expanding and sliding windows, as well as custom validation splits.
User-Friendly API: Designed with ease of use in mind, Skforecast’s API simplifies setup and customization.
Steps for Effective Backtesting with Skforecast
To ensure effective backtesting, follow these steps for setup and implementation:
Install Skforecast and Dependencies:
pip install skforecast
Data Preparation and Formatting: Time series data should be indexed by date and formatted as a pandas DataFrame. Here’s an example of setting up the data:
import pandas as pd
data = pd.read_csv('time_series_data.csv', index_col='date', parse_dates=True)
Choose a Forecasting Model: Skforecast allows you to choose from various scikit-learn regressors. For example, here’s how to set up a linear regression model with Skforecast:
from sklearn.linear_model import LinearRegression
from skforecast.ForecasterAutoreg import ForecasterAutoreg
forecaster = ForecasterAutoreg(
regressor=LinearRegression(),
lags=12
)
Select a Backtesting Strategy:
Expanding Window: Ideal for stable data, where you progressively expand the training set.
Sliding Window: Keeps a fixed training size and slides over time, effective for data with seasonality.
To get accurate results, it’s essential to select the right backtesting technique and follow best practices.
Expanding Window Technique
The expanding window method is well-suited for data without drastic changes over time. With each iteration, the training set expands to include more data, improving the model’s ability to capture patterns.
The sliding window method maintains a consistent training set size, moving forward with each new data point. This technique is useful for time series data with changing patterns, as it always reflects the most recent data.
Cross-Validation for Time Series
To reduce variance and improve model reliability, cross-validation techniques adapted for time series can be used. These methods involve multiple overlapping training/test splits, providing a more comprehensive evaluation of model performance.
Implementing Backtesting with Skforecast: A Step-by-Step Example with Advanced Techniques
Let’s expand our example using Skforecast with a more complex regressor, like a Random Forest, which can capture non-linear relationships in data.
Troubleshooting Common Backtesting Issues with Skforecast
In real-world applications, you may encounter issues such as data leakage, overfitting, or insufficient data. Here are some solutions:
Data Leakage: Ensure no future data is used in training by properly setting up your training/test splits.
Overfitting: Avoid using overly complex models unless you have sufficient data to prevent overfitting. Simpler models like linear regression can often perform adequately.
Insufficient Data: For limited datasets, consider using cross-validation to maximize data usage, but beware of compromising temporal sequence integrity.
Skforecast vs. Alternatives: Prophet, ARIMA, and statsmodels
While Skforecast is effective for autoregressive models, other libraries offer specific strengths:
Prophet
Prophet is a powerful tool ideal for seasonal data, especially those with daily or hourly cycles. It’s designed to handle seasonality with minimal user input, automatically detecting and integrating holidays into its forecasting model. This makes Prophet well-suited for applications such as retail sales or website traffic forecasting, where seasonal trends and holiday effects play significant roles.
ARIMA/Statsmodels
ARIMA, available through the Statsmodels library, excels with stationary data—where mean and variance remain constant over time. ARIMA is a foundational method for time series forecasting, but when data shows seasonal patterns, the SARIMA (Seasonal ARIMA) model adds additional layers, improving forecast accuracy for seasonal datasets. These models are ideal for situations with stable trends and predictable periodic changes.
LSTMs (via Keras)
Long Short-Term Memory (LSTM) networks, a type of recurrent neural network available via Keras, are effective for complex, non-linear time series data. They can capture intricate dependencies and patterns across time, making them suitable for highly dynamic datasets such as stock prices or energy consumption. LSTMs require large datasets and substantial computational power, but they excel at recognizing long-term dependencies and trends that simpler models might overlook, making them invaluable for sophisticated forecasting tasks.
Comparison Table:
Feature
Skforecast
Prophet
ARIMA/Statsmodels
LSTMs (Keras)
Ideal Data Type
Univariate, Linear
Seasonal, Trend
Stationary
Non-linear, Complex
Ease of Use
High
Medium
Medium
Low
Flexibility
High (Custom Models)
Medium
High
High
Model Complexity
Moderate
Low
Low-Medium
High
Case Study: Using Skforecast for Demand Forecasting
To illustrate Skforecast’s capabilities, let’s look at a retail demand forecasting case:
Problem: A retail chain wants to forecast weekly demand for a specific product.
Data: Three years of historical sales data.
Objective: Minimize forecast error and ensure stock levels meet demand.
Implementation Steps:
Load and Preprocess Data: Convert the dataset into a time series format indexed by weeks.
Select Model: Use a Gradient Boosting Regressor, which handles seasonality well.
Backtest and Evaluate: Split the data by year to perform year-over-year backtesting.
Optimization: Use grid search with cross-validation to fine-tune hyperparameters.
Results: Evaluate using MAE to measure forecasting accuracy. Adjust model parameters based on error analysis.
Future Trends in Backtesting and Time Series Forecasting
As AI and machine learning technologies evolve, we see a shift towards more automated solutions for time series forecasting:
AutoML: Platforms like H2O.ai offer automated feature engineering, model selection, and hyperparameter tuning for time series.
Deep Learning: Architectures like Transformers, originally developed for NLP, are now adapted for time series to capture complex dependencies.
Explainable AI: More tools are focusing on interpretability, allowing users to understand how their forecasts are generated.
Backtesting with Skforecast offers a powerful means to validate time series forecasting models, enabling a thorough assessment of model performance across various scenarios. Skforecast provides essential functionality for time series forecasting, supporting excellent regression algorithms, exponential smoothing, and robust workflows. It allows for the inclusion of exogenous variables, which can improve model accuracy by accounting for external factors affecting trends.
A wide range of forecasting strategies is available within Skforecast, including simple model setups, hierarchical forecasts, and gradient boosting integration for advanced applications. Using Skforecast, you can easily perform batch forecast evaluations, plot predictions, and create hierarchical forecasts tailored to complex datasets. This flexibility supports both basic forecasting workflows and more complex operations like traffic prediction with machine learning.
By leveraging Skforecast’s advanced capabilities, such as handling time series differentiation and prediction intervals, you can develop sophisticated models that handle diverse business needs. The tool’s ability to handle gradient boosting for trend modeling, along with support for fields of statistics, enables it to cater to various use cases like forecasting web traffic or financial trends. As Skforecast continues to evolve, it remains a valuable resource for modeling time series trends, offering users an effective way to explore prediction methodologies, avoid pitfalls like false differentiation, and integrate advanced statistical techniques into their forecasting practices.
References
Hyndman, Rob J., and George Athanasopoulos. Forecasting: Principles and Practice. 3rd ed., OTexts, 2018. Available online at https://otexts.com/fpp3/.
James, Gareth, et al. An Introduction to Statistical Learning: With Applications in R. 2nd ed., Springer, 2021.
Sandanayake, Kanchana. “A Guide to Backtesting: What It Is and How to Get Started.” Skforecast Documentation, Skforecast, 2024, https://skforecast.org/.
Choudhury, Soumyajit, and J. Cristian Bota. “Comparative Study of Statistical and Machine Learning Models for Time Series Forecasting.” Journal of Business Research, vol. 145, 2023, pp. 1-13.
Taylor, Sean J., and Benjamin Letham. “Prophet: Forecasting at Scale.” Journal of Open Source Software, vol. 1, no. 1, 2017, p. 3. https://doi.org/10.21105/joss.00013.
Adhikari, Ratnadip, and R. K. Agrawal. An Introductory Study on Time Series Modeling and Forecasting. Springer, 2013.
Introduction to Backtesting in Forecasting
This is a comprehensive guide to backtesting with skforecast in python with examples. Backtesting in time series forecasting simulates how a model would perform if it had been applied in the past, based on historical data. This process is crucial for validating models before deployment, as it offers insights into potential forecasting errors and helps optimize model performance. By testing how models perform under different scenarios, forecasters can better understand limitations and avoid overfitting.
For instance, backtesting allows you to assess how a model might perform during different seasonal cycles or economic conditions, enabling a more robust evaluation. In this guide, we’ll leverage Skforecast, a Python library that integrates seamlessly with scikit-learn, to facilitate backtesting for time series models.
Table of contents
What is Backtesting?
Backtesting is a validation technique in time series analysis that evaluates the accuracy of forecasting models by testing them on historical data. In forecasting web traffic, backtesting methods help refine predictions and improve model reliability by revealing how they would perform in real-world scenarios. This process is crucial for both web traffic and financial backtesting, as it provides insights into time series trends and supports robust prototyping.
Utilizing a variety of forecaster types and validation strategies, backtesting in time series forecasting often involves tools for training models with machine learning, enhancing predictions with a range of functionalities. Effective backtesting captures the intricacies of time series data, addressing essential aspects such as seasonal variations and feature requests for more specific trends. By modeling time series trends with these advanced techniques, analysts can ensure their models are both accurate and adaptable to changing patterns, making backtesting a core component of forecasting methodologies.
Also Read: How is AI Improving Weather Forecasting?
Examples of Backtesting
Backtesting is widely used across various fields, including intermittent demand forecasting, where it assesses models that predict irregular or sporadic demand patterns. For instance, forecasting energy demand requires handling exogenous variables like weather conditions or public holidays, which significantly impact usage. By incorporating series differentiation and using recursive multi-step forecasting, models can accommodate the complexities of demand with machine learning, improving accuracy. Forecaster objects and time series differentiation enable more precise modeling, while exogenous features capture additional influences that affect energy consumption.
In the context of PV (photovoltaic) power forecasting, backtesting aids in building interpretable forecasting models that can predict power demand and output. Multi-series forecasting is especially relevant here, as energy production data often involve multiple time series, such as different geographic regions or types of PV systems. Techniques like direct multi-step forecasting are used to generate probabilistic forecasts, which provide a range of possible outcomes. These forecasts, combined with quantile regression, offer robust estimates of PV output under various conditions, making the models more resilient to data variability.
For more advanced applications, such as forecasting energy demand with machine learning, a hyperparameter search via random search or k-fold cross-validation fine-tunes model performance. Tools like quantile forecasts and categorical features help produce probabilistic forecasts, giving a more comprehensive view of potential future states. These methods support both single and multi-step forecasters, allowing analysts to explore the implications of different forecast periods. With interpretable forecasting models and robust backtesting techniques, forecasters can better handle the intricacies of time series data and produce reliable forecasts that are crucial for effective decision-making.
What is Skforecast?
Skforecast simplifies time series forecasting by automating tasks like lag creation and offering a wide selection of models through scikit-learn. This tool is suitable for both beginners and advanced users interested in performing autoregressive forecasting and backtesting due to its intuitive interface and support for custom model integrations.
Skforecast’s primary functionality centers on the
ForecasterAutoreg
class, which is pivotal for implementing autoregressive forecasting models. This class is effective for applications like demand forecasting and stock price prediction by incorporating lag values. Lags allow models to incorporate past values into future forecasts, enhancing accuracy and predictive power. The development of Skforecast focuses on making forecasting accessible while enabling complex forecasting strategies such as recursive and direct approaches.The recursive strategy, or series recursive, uses previous forecasted values as inputs for future predictions, which is ideal for capturing ongoing patterns but may propagate errors. The direct strategy, on the other hand, forecasts each time step individually, offering a realistic assessment for multi-step forecasts, reducing cumulative error. By combining Skforecast with gradient boosting models, users can achieve robust forecasting and obtain prediction intervals to gauge forecast certainty. This range of functionalities makes Skforecast an adaptable tool for numerous real-world forecasting tasks.
Key Features of Skforecast
Steps for Effective Backtesting with Skforecast
To ensure effective backtesting, follow these steps for setup and implementation:
Install Skforecast and Dependencies:
Data Preparation and Formatting: Time series data should be indexed by date and formatted as a pandas DataFrame. Here’s an example of setting up the data:
Choose a Forecasting Model: Skforecast allows you to choose from various scikit-learn regressors. For example, here’s how to set up a linear regression model with Skforecast:
Select a Backtesting Strategy:
Common Backtesting Techniques and Best Practices
To get accurate results, it’s essential to select the right backtesting technique and follow best practices.
Expanding Window Technique
The expanding window method is well-suited for data without drastic changes over time. With each iteration, the training set expands to include more data, improving the model’s ability to capture patterns.
Sliding Window Technique
The sliding window method maintains a consistent training set size, moving forward with each new data point. This technique is useful for time series data with changing patterns, as it always reflects the most recent data.
Cross-Validation for Time Series
To reduce variance and improve model reliability, cross-validation techniques adapted for time series can be used. These methods involve multiple overlapping training/test splits, providing a more comprehensive evaluation of model performance.
Implementing Backtesting with Skforecast: A Step-by-Step Example with Advanced Techniques
Let’s expand our example using Skforecast with a more complex regressor, like a Random Forest, which can capture non-linear relationships in data.
Step 1: Set Up the Random Forest Regressor
Step 2: Apply Sliding Window with Multiple Steps Ahead
Instead of forecasting one step at a time, you can adjust Skforecast to predict multiple steps ahead.
Step 3: Visualize the Results
Visualization helps in understanding model performance over time. Here’s how to plot actual vs. predicted values:
Troubleshooting Common Backtesting Issues with Skforecast
In real-world applications, you may encounter issues such as data leakage, overfitting, or insufficient data. Here are some solutions:
Skforecast vs. Alternatives: Prophet, ARIMA, and statsmodels
While Skforecast is effective for autoregressive models, other libraries offer specific strengths:
Prophet
Prophet is a powerful tool ideal for seasonal data, especially those with daily or hourly cycles. It’s designed to handle seasonality with minimal user input, automatically detecting and integrating holidays into its forecasting model. This makes Prophet well-suited for applications such as retail sales or website traffic forecasting, where seasonal trends and holiday effects play significant roles.
ARIMA/Statsmodels
ARIMA, available through the Statsmodels library, excels with stationary data—where mean and variance remain constant over time. ARIMA is a foundational method for time series forecasting, but when data shows seasonal patterns, the SARIMA (Seasonal ARIMA) model adds additional layers, improving forecast accuracy for seasonal datasets. These models are ideal for situations with stable trends and predictable periodic changes.
LSTMs (via Keras)
Long Short-Term Memory (LSTM) networks, a type of recurrent neural network available via Keras, are effective for complex, non-linear time series data. They can capture intricate dependencies and patterns across time, making them suitable for highly dynamic datasets such as stock prices or energy consumption. LSTMs require large datasets and substantial computational power, but they excel at recognizing long-term dependencies and trends that simpler models might overlook, making them invaluable for sophisticated forecasting tasks.
Comparison Table:
Case Study: Using Skforecast for Demand Forecasting
To illustrate Skforecast’s capabilities, let’s look at a retail demand forecasting case:
Implementation Steps:
Future Trends in Backtesting and Time Series Forecasting
As AI and machine learning technologies evolve, we see a shift towards more automated solutions for time series forecasting:
Also Read: The role of AI in vaccine distribution
Conclusion
Backtesting with Skforecast offers a powerful means to validate time series forecasting models, enabling a thorough assessment of model performance across various scenarios. Skforecast provides essential functionality for time series forecasting, supporting excellent regression algorithms, exponential smoothing, and robust workflows. It allows for the inclusion of exogenous variables, which can improve model accuracy by accounting for external factors affecting trends.
A wide range of forecasting strategies is available within Skforecast, including simple model setups, hierarchical forecasts, and gradient boosting integration for advanced applications. Using Skforecast, you can easily perform batch forecast evaluations, plot predictions, and create hierarchical forecasts tailored to complex datasets. This flexibility supports both basic forecasting workflows and more complex operations like traffic prediction with machine learning.
By leveraging Skforecast’s advanced capabilities, such as handling time series differentiation and prediction intervals, you can develop sophisticated models that handle diverse business needs. The tool’s ability to handle gradient boosting for trend modeling, along with support for fields of statistics, enables it to cater to various use cases like forecasting web traffic or financial trends. As Skforecast continues to evolve, it remains a valuable resource for modeling time series trends, offering users an effective way to explore prediction methodologies, avoid pitfalls like false differentiation, and integrate advanced statistical techniques into their forecasting practices.
References
Hyndman, Rob J., and George Athanasopoulos. Forecasting: Principles and Practice. 3rd ed., OTexts, 2018. Available online at https://otexts.com/fpp3/.
James, Gareth, et al. An Introduction to Statistical Learning: With Applications in R. 2nd ed., Springer, 2021.
Sandanayake, Kanchana. “A Guide to Backtesting: What It Is and How to Get Started.” Skforecast Documentation, Skforecast, 2024, https://skforecast.org/.
Choudhury, Soumyajit, and J. Cristian Bota. “Comparative Study of Statistical and Machine Learning Models for Time Series Forecasting.” Journal of Business Research, vol. 145, 2023, pp. 1-13.
Taylor, Sean J., and Benjamin Letham. “Prophet: Forecasting at Scale.” Journal of Open Source Software, vol. 1, no. 1, 2017, p. 3. https://doi.org/10.21105/joss.00013.
Adhikari, Ratnadip, and R. K. Agrawal. An Introductory Study on Time Series Modeling and Forecasting. Springer, 2013.
Share this: