Time Series Forecast Error

We use the following terminology: if y1, …, yn represents a time series, then ŷi represents the ith forecasted value, where i ≤ n. For i ≤ n, the ith error ei (aka residual) is then

image001z

Our goal is to find a forecast that minimizes the errors.

Key measures

A number of measures are commonly used to determine the accuracy of a forecast. These include the mean absolute error (MAE), mean squared error (MSE) and root mean squared error (RMSE).

image110x

Note that MAE is also commonly called mean absolute deviation (MAD). This version of MAD should not be confused with the median absolute deviation (MAD) described in Measures of Variability.

Other measures

Some other measurements are mean absolute percentage error (MAPE), mean absolute scaled error (MASE) and symmetric mean absolute percentage error (SMAPE).

image117c

For data with seasonality (see Holt-Winter Forecasting) where the periodicity is c, the formula for MASE becomes

image118c

Theil statistics

Finally, there is Theil’s U statistic, which is computed by the formula

Theil's U2 statistic

If U < 1, the forecasting technique is better than guessing. If U = 1 then the forecasting technique is as good as guessing. If U > 1 then the forecasting technique is worse than guessing.

Actually, U is also called Theil’s U2 statistic. There is also a less-often used U1 statistic

Theil's U1 statisticU1 takes values between 0 and 1, with values nearer to 0 representing greater forecast accuracy.

Real Statistic Support

Click here for information about Real Statistics support for the forecast error measures described on this webpage.

Examples Workbook

Click here to download the Excel workbook with the examples described on this webpage

References

Hyndman, R. J. and Athanasopoulos, G. (2018) Evaluating forecast accuracy. Forecasting: Principles and Practice. 2nd ed.
https://otexts.com/fpp2/accuracy.html

Oracle (2018) Time-series forecasting error measures
https://docs.oracle.com/en/cloud/saas/planning-budgeting-cloud/pfusu/time-series_forecasting_error_measures.html

9 thoughts on “Time Series Forecast Error”

  1. Dear Dr. Zaiontz,

    1. Is the coefficient of variability (Std. Deviation/Average sale) a good measure of forecast accuracy?
    2. Will it be possible to demonstrate how to determine the smoothing factor (alpha) using 12-month history of original forecast versus actual sale?

    Thanks.

    Aftab

    Reply
    • Hello Aftab,
      1. It depends on what data set you are calculating the coefficient of variability on. Taking the coefficient of variability on the original time series is not a useful measure of forecast accuracy.
      2. Which forecasting method do you have in mind? If Exponential Smoothing, this is explained on this website.
      Charles

      Reply
  2. Hello Dr Zaiontz,

    I’m building a proof-of-concept forecasting tool in Excel that helps our business to select the best possible model. The performance metric I would like to use is the average relative MAEs using weighted geometric mean (AvgRelMAE) (Davydenko, A., & Fildes, R. (2016))

    Is this included somehow in RealStats?

    Thank you,
    Steven

    Reply
    • Hello Steven,
      Real Statistics will use the MSE and MAE metrics, but not the AvgRelMAE metric. You can, however, build the models as shown on the website using Solver but instead of using the MSE or MAE metric, you can use the AvgRelMAE metric.
      Charles

      Reply
    • The implementation of AvgRelMAE is straightforward:

      1) create a table with the following columns:
      MAE(method1) MAE(method2) …. MAE(methodN)

      2) select your benchmark method (say, method1)

      3) calculate RelMAEs:
      RelMAE(method2)=MAE(method2)/MAE(method1)
      RelMAE(method3)=MAE(method3)/MAE(method1)

      4) calculate logs of RelMAEs, logRelMAE

      5) calculate arithmetic means of logRelMAE, mean(logRelMAE)

      6) calculate exp(mean(logRelMAE)) for each method

      Reply
  3. Hi DR.
    Need help on RMSE values.
    How can I say it’s small or not?
    Compared to mean error or to what?
    Is their rules to interpret RMSE?
    Thanks

    Reply

Leave a Comment