RESULTS OF A SPECIAL QUESTIONNAIRE

FOR PARTICIPANTS IN THE ECB SURVEY

OF PROFESSIONAL FORECASTERS 1

This document provides a summary of the 1 FREQUENCY OF UPDATES OF THE FORECASTS

aggregate results of a special questionnaire REPORTED IN THE SPF
which was sent to the participants in the ECB
Survey of Professional Forecasters (SPF) in The majority of respondents (84%) reported
autumn 2008, in the context of the ten-year that their forecasts are updated on a regular
anniversary of the SPFs launch in January calendar basis (see Chart 1). Around
1999.2 one-third indicated that they update their
forecasts following data releases or other
There were 45 replies to the special events (such as the current financial market
questionnaire, which is approximately three- turmoil). In addition, a number of respondents
quarters of the average number of responses commented that sometimes they may also
received in the regular survey rounds update forecasts in the face of significant
(around 60). The respondents were a broadly shocks. Taking these comments into account,
representative cross section of the SPF panel in it would seem that a substantial proportion of
terms of type of forecaster. The questionnaire respondents (also around one-third) update
contained questions on timeliness and
methods of forecasting, in particular relating
to the frequency of forecast updates, the
1 An abridged summary of this note was published as a
models used, the importance of judgement, box in the April 2009 issue of the ECB Monthly Bulletin.
the generation of reported probability We would like to acknowledge the useful comments and
suggestions on earlier drafts of the questionnaire received
distributions and assumptions regarding other from Kenneth Wallis (University of Warwick), Tom Stark
variables implicitly underlying the forecasts (Federal Reserve Bank of Philadelphia) and Aurelio Maccario
(see Annex 1). (UniCredit). Any questions or queries on the questionnaire
and results should be addressed to Aidan Meyler or
Ieva Rubene at ecb-spf@ecb.europa.eu.
It should be noted that on some occasions the 2 Individual answers are confidential.
percentages reported may add up to more than
100%, as some respondents indicated more
Chart 1 When do you update your forecasts?
than one category. Tables reporting the replies
and the response rates for each question are
available in Annex 2. (percentages)

100 100
In summary, the results show that the SPF
80 80
responses are quite timely and that the forecasts
are based on heterogeneous assumptions that are 60 60
predominantly generated in house. In addition,
40 40
although both structural and time series models
are widely used, judgement also plays a key 20 20
role, in particular for the reported probability
0 0
distributions. It is thus very important to calendar driven data dependent
consider the heterogeneity of the SPF forecasts
Source: ECB.
when analysing and interpreting the results of Note: Adds up to more than 100% as some respondents selected
both categories.
the SPF.

ECB
Results of a special SPF questionnaire
April 2009 1
 Chart 2 If it is calendar driven, how often do while two respondents reported that they update
you update your forecasts? them continuously.

(percentages)
Most respondents indicated that they provide
60 60 their latest available forecast in each SPF round,
50 50 with only a small proportion preparing a new
40 40 forecast for the SPF (see Chart 3). However, a
number of respondents (27%) said that they may
30 30
partially update their forecasts when responding
20 20 to the SPF. Overall, given the high frequency of
10 10 regular updates and respondents comments to the
0 0 effect that they also adjust their forecasts or prepare
quarterly monthly continuously other new ones in exceptional circumstances, the replies
Source: ECB. suggest that the SPF responses are quite timely.
Note: Adds up to more than 100% as some respondents selected
more than one category.

2 FREQUENCY OF THE DATA BEING FORECAST

their forecasts on a calendar basis and also on an
event or data-driven basis, depending on the Most respondents generally follow the frequency
specific circumstances. of the data series they forecast. Hence, most
panellists forecast HICP inflation at a monthly
Of those respondents who update their forecasts frequency, with GDP being forecast at a
regularly according to a calendar, over 50% quarterly frequency (see Chart 4). The responses
reported that their forecasts are updated on a regarding unemployment forecasts were less
quarterly basis, with a slightly smaller share homogenous, with some respondents forecasting
(35%) updating them on a monthly basis at a monthly frequency and others forecasting
(see Chart 2). A small proportion (around 10%) at a quarterly frequency. A small number of
reported that they update their forecasts less respondents also reported that their longer-term
frequently (e.g. two or three times per year), forecasts are only at an annual frequency.

Chart 3 When responding to the SPF, what Chart 4 What is the highest frequency of
forecast do you provide? data at which you model and forecast?

(percentages) (percentages)

70 70 70 70
60 60 60 60
50 50 50 50
40 40 40 40
30 30 30 30
20 20 20 20
10 10 10 10
0 0 0 0
latest available new forecast it depends monthly quarterly annual it depends

Source: ECB. Source: ECB.

Note: Many of those providing qualitative information indicated Note: Many respondents reported that they follow the frequency
that they may partially update their forecasts when responding of the underlying variable being forecast (i.e. monthly for
to the SPF, if changes are significant enough to warrant it. inflation and quarterly for GDP).

ECB
Results of a special SPF questionnaire
2 April 2009
3 FORECASTING TECHNIQUES AND MODELS Chart 5 Degree of judgement applied to
forecasts
3.1 RELATIVE WEIGHTS OF JUDGEMENT AND
(percentages)
MODELS
short-term forecasts
medium-term forecasts
SPF participants were asked to what extent long-term forecasts
their forecasts are based on a model and to 60 60
what extent they are based on their judgement.
50 50
The forecasters were encouraged to assign
approximate weights (percentages) for the 40 40
importance of models and of judgement.
30 30
Regarding the use of models for forecasting,
the questionnaire suggested time series models 20 20
(including auto regressive integrated moving
10 10
average (ARIMA), single equation, vector auto
regression (VAR) or vector error correction 0 0
HICP inflation GDP growth unemployment rate
(VEC), and factor models), traditional supply
and demand-based macro models and dynamic Source: ECB.

stochastic general equilibrium (DSGE) models.

Some respondents indicated that their forecasts With respect to the overall importance of
are 100% judgement-based (see Table 1). judgement in the forecasts of the SPF
However, analysis of the qualitative answers participants, the answers show that, on average,
suggests that there may be two types of respondents consider their forecasts to be 40%
behaviour behind these responses. First, there judgement-based (see Chart 5).3 There are no
are forecasters who, for certain variables or major differences across variables or horizons,
forecast horizons, do not use models but report with the exception of inflation. Judgement
a forecast based on their judgment. Second, there applied to short-term inflation forecasts (up to
are forecasters who use models, but report the one year ahead) is indicated to be around 37%,
final outcome as 100% judgement-based because increasing to 54% for longer-term forecasts
their initial model-based forecast can be adjusted (five years ahead). For real GDP forecasts,
in any direction and to any extent. judgement, on average, has slightly less weight
than for unemployment rate forecasts, and for
It is not possible to distinguish between these both variables there are no significant
two groups. Therefore, when considering the differences across forecast horizons, although
relative weights of model-based and judgement- there is a small increase for the longer horizon.
based forecasts, one should keep in mind
that the results may be skewed in favour of 3 Excluding those who report their forecasts to be 100%
judgement. judgement-based, the percentage is 33%.

Table 1 Number of respondents reporting 100% judgement-based forecasting

HICP inflation GDP growth Unemployment rate

Short-term forecasts 5 5 8
Medium-term forecasts 6 6 7
Long-term forecasts 10 8 9
Source: ECB.

ECB
Results of a special SPF questionnaire
April 2009 3
 Chart 6 Relative weights of model-based and Chart 7 What types of model do you use for
judgement-based forecasting forecasting?

(percentages) (percentages)

short-term forecasts 80 80
medium-term forecasts
70 70
long-term forecasts
60 60
HICP inflation 50 50

60 60 40 40
30 30
50 50 20 20
10 10
40 40
0 0
1 2 3 4 5 6 7
30 30
1 Time series (ARIMA)
2 Time series (single equation)
20 20 3 Time series (VAR/VEC)
4 Time series (other, e.g. factor models)
10 10 5 Macro (DSGE)
6 Macro (traditional or other)
0 0 7 Other, please specify
macro - macro - time judgement other,
Source: ECB.
traditional DSGE series please
specify

GDP growth 3.2 TYPES OF MODEL USED FOR FORECASTING

60 60
The responses indicated that the type of
50 50 model preferred varies according to the
40 40 forecast horizon and to the variable being
forecast. A pattern emerged where the use
30 30
of time series models is more common for
20 20 shorter-term horizons and for inflation forecasts,
whereas traditional supply and demand-based
10 10
macro models are used more for longer-term
0 0 horizons and slightly more for real GDP and
macro - macro - time judgement other,
traditional DSGE series please
unemployment rate forecasting (see Chart 6).
specify

Unemployment rate Considering in more detail the types of model

used for forecasting, most respondents (around
60 60
85%) reported that they use at least one type of
50 50 time series model. Three of these are relatively
widely used: ARIMA, single equation, and
40 40
VAR or VEC models (see Chart 7). A smaller
30 30 proportion uses other time series models, such as
factor models. Most respondents who use time
20 20
series models reported that they use two or
10 10 more types of such models. Almost 70% of
respondents reported that they use traditional
0 0
macro - macro - time judgement other, supply and demand-based macro models, while
traditional DSGE series please
specify very few forecasters indicated that they use
Source: ECB.
DSGE models or some other type of model not
specified in the questionnaire.

ECB
Results of a special SPF questionnaire
4 April 2009
 Chart 8 How are your assumptions for oil Chart 10 How are your assumptions for
prices derived? interest rates derived?

(percentages) (percentages)

100 100 100 100

80 80 80 80

60 60 60 60

40 40 40 40

20 20 20 20

0 0 0 0
in house markets consensus other in house markets consensus other

Source: ECB. Source: ECB.

Note: Adds up to more than 100%, as some respondents selected Note: Adds up to more than 100%, as some respondents selected
more than one category. more than one category.

4 OTHER VARIABLES AND CONDITIONING futures or spot rates. A few respondents

ASSUMPTIONS reported that they use external forecasts
to complement and cross-check in-house
With regard to other variables and conditioning forecasts. In terms of other sources, a small
assumptions, most respondents produce in-house number of respondents use automatic rules (e.g.
forecasts for oil prices, exchange rates, a random walk). These replies suggest that, as
interest rates and wage growth (see Charts is always highlighted in the reporting of the
8-11). In-house forecasts of oil prices are often SPF results, SPF responses reflect a relatively
complemented by market data, for example diverse set of views and assumptions.

Chart 9 How are your assumptions for Chart 11 How are your assumptions for wage
exchange rates derived? growth derived?

(percentages) (percentages)

100 100 100 100

80 80 80 80

60 60 60 60

40 40 40 40

20 20 20 20

0 0 0 0
in house markets consensus other in house consensus other

Source: ECB. Source: ECB.

Note: Adds up to more than 100%, as some respondents selected Note: Adds up to more than 100%, as some respondents selected
more than one category. more than one category.

ECB
Results of a special SPF questionnaire
April 2009 5
 5 PROBABILITY DISTRIBUTIONS AND instead of the mean, for example if warranted
CORRESPONDING POINT ESTIMATES in the light of the economic environment.

The respondents were also asked how The answers to the question about the
they generate their reported probability interpretation of the intervals for the probability
distributions for HICP inflation, GDP growth distributions in the SPF have important
and the unemployment rate. A large majority implications for assessing the balance of risks
of respondents (79%) said that these probability to the point forecast. It also has implications
distributions are estimated on the basis of for fitting a continuous probability distribution
judgement, while 17% generate them from to the discrete distributions provided by
models (see Chart 12). A small number reported the respondents and the interpretation of
that the probability distributions provided in the results. To the question on how the SPF
the SPF are based on a functional form (usually panellists interpret the interval from 1.5%
the standard normal distribution). to 1.9%, a majority (76%) indicated that they
use a standard rounding convention, i.e. the
Forecasters were also asked whether they report interval is assumed to be 1.45% to 1.95%,
their mean, modal or median forecast.4 The while 17% of the respondents reported that
replies to this question revealed that a clear they interpret it to be from 1.50% to 1.99%
majority of respondents (75%) provide the point (or to 2.0% with the end point not included;
estimate which corresponds to the mean of their
reported probability distribution (see Chart 13).
Almost 20% reported that their point forecast
corresponds to the median, while a small 4 The mean is the weighted average of all possible outcomes,
where the weights are the respondents assessments of the
proportion (7%) indicated that it corresponds to probability associated with each outcome. The mode is the
the mode of their reported probability forecast that is assessed to be most likely to occur, but does
distribution. A few respondents indicated that not necessarily reflect the balance of risks surrounding the
most likely outcome. The median is the outcome with 50%
they may deviate from the measure that they probability above and 50% probability below, and does not
usually report using the mode or median take into account the outliers above or below the median.

Chart 12 How do you generate your reported Chart 13 Does your reported forecast refer
probability distribution? From to the mean, mode or median of your
reported probability distribution?
(percentages) (percentages)

80 80 80 80

60 60 60 60

40 40 40 40

20 20 20 20

0 0 0 0
model functional form judgement mean mode median

Source: ECB. Source: ECB.

Note: Adds up to more than 100%, as one respondent selected Note: Adds up to more than 100%, as some respondents selected
more than one category. more than one category.

ECB
Results of a special SPF questionnaire
6 April 2009
 Chart 14 How do you interpret the
interval of 1.5% - 1.9%?

1.40% - 1.90%
(7%)

1.45% - 1.95%
(17%)

1.50% - 2.00%
(76%)

Source: ECB.

see Chart 14). A few respondents indicated that

they treat the interval as being from 1.40% to
1.90%.

6 EXTERNAL USE OF THE FORECASTS

Approximately two-thirds of respondents

(65%) stated that they publish externally the
forecasts they send to the ECB when replying
to the SPF.

ECB
Results of a special SPF questionnaire
April 2009 7
 ANNEX 1

SPECIAL QUESTIONNAIRE FOR PARTICIPANTS IN

THE ECB SURVEY OF PROFESSIONAL FORECASTERS 1

A) QUESTIONS ON FORECASTING

1a. When do you update your forecasts? Do you always follow a regular calendar schedule or
do you sometimes update in the light of significant shocks (e.g. food / oil price) or new data
releases?
Calendar-driven Data dependent Other (please explain)

1b. If it is calendar driven, how often do you update your forecasts?

Quarterly Monthly Continuously Other (please explain)

1c. When responding to the SPF do you

provide your latest available forecast?
prepare a new forecast?
it depends on the timing
Additional comments:

2. What is the highest frequency of data at which you model/forecast? If this varies
systematically across forecast variables or horizons please elaborate.
a) monthly
b) quarterly
c) annual
d) depends on the variable/horizon
Additional comments:

1 The questionnaire was sent out on 18 September 2008 and the SPF participants were asked to return the form by 15 October. Some
replies were received after the deadline. The questionnaire also contained a few internal and procedural questions, which aimed to
elicit feedback about the SPF itself. These are excluded from this sample form.

ECB
Results of a special SPF questionnaire
8 April 2009
3. To what extent (percentage) are your forecasts model- or judgement-based? If this
varies systematically across forecast variables and horizons please elaborate below.
We recognise that this can just be an approximation and may vary over time; nonetheless
it is of interest to know how forecasts are generated. Note: time series models include
ARIMA, single equation, VAR, factor, etc. models.

3a. Shorter-term (one year or less)

HICP GDP Unemployment
Macro-Model Traditional
DSGE
Time series
Judgement
Other,
please specify
100% 100% 100%

3b. Medium-term (up to two years)

HICP GDP Unemployment
Macro-Model Traditional
DSGE
Time series
Judgement
Other,
please specify
100% 100% 100%

3c. Longer-term (five years ahead)

HICP GDP Unemployment
Macro-Model Traditional
DSGE
Time series
Judgement
Other,
please specify
100% 100% 100%

Additional comments:

ECB
Results of a special SPF questionnaire
April 2009 9
 4. What type of models do you use and what weight do you attach to them?
E.g. Single-equation, Macro model, VAR, DSGE, etc. If this varies systematically across
forecast variables and horizons please elaborate. We recognise that this can just be an
approximation;
Time series - ARIMA %
Single equation %
VAR/VEC %
Other (e.g. factor models) %

Macro-Economic - DSGE %
Traditional, other %
Other, please specify - %

Additional comments:

5. If you provide information on the external assumptions (i.e. oil prices, exchange rates,
interest rates and wage growth), how do you form these?
Oil prices:
in-house forecast futures/market prices consensus/average forecast
other (please explain)
Exchange rates:
in-house forecast futures/market prices consensus/average forecast
other (please explain)
Interest rates:
in-house forecast futures/market prices consensus/average forecast
other (please explain)
Wage growth:
in-house forecast consensus/average forecast
other (please explain)

ECB
Results of a special SPF questionnaire
10 April 2009
6. Does your reported point forecast refer to the mean, mode or median of your reported
probability distribution in the SPF?
Mean Mode Median Other
We do not estimate probability distributions
Additional comments:

7. How do you calculate your reported probability distributions? If this varies

systematically across forecast variables and horizons please elaborate.
a) Derived from a model
If they are derived from the standard errors of a model are they based on a specific
assumption about the functional form of the distribution? If yes, then what is this form?

b) Based on a functional form (normal, skew, etc.) with key parameters (e.g. standard deviation,
skew) selected outside the model framework
If yes, then what is this form?

c) Judgement-based
Additional comments:

8. In the ECB SPF respondents are asked to assign probabilities associated with an outcome
occurring within given ranges (e.g. 1.5%-1.9%, 2.0%-2.4%, etc.). These ranges are open to
interpretation which sometimes makes it difficult to assess the balance of risks to the point
forecast. How do you interpret the interval of 1.5%-1.9%?
i) 1.40%-1.90%
ii) 1.45%-1.95%
iii) 1.50%-2.00%

9. Do you publish externally the forecasts you send to us?

Yes No
Additional comments:

ECB
Results of a special SPF questionnaire
April 2009 11
 ANNEX 2

SPECIAL QUESTIONNAIRE FOR PARTICIPANTS

IN THE ECB SURVEY OF PROFESSIONAL
FORECASTERS
SUMMARY OF THE ANSWERS AND THE RESPONSE RATES1

Q1a When do you update your forecasts?

Do you always follow a regular calendar schedule or do you sometimes update them in
the light of significant shocks (e.g. food or oil price shocks) or new data releases?
Calendar driven 37 84%
Data dependent 13 30%
Responses 44
Note: A closer reading of the 16 qualitative responses would suggest that an additional
nine responses that were provided could also be classified as data dependent as
forecasters commented that they would update their forecasts in the face of significant
shocks which would bring the number to 22. Based on both the explicit and the
qualitative answers of respondents, 15 forecasts (or one-third) appear to be both
calendar and data driven.

Q1b If it is calendar driven, how often do you update your forecasts?

Quarterly 19 51%
Monthly 13 35%
Continuously 2 5%
Other, please explain 4 11%
Responses 37
Note: One respondent indicated both quarterly and monthly. Four respondents indicated
that their forecast updates are less frequent than quarterly normally two to three times
per year.

Q1c When responding to the SPF do you

provide your latest available forecast 29 66%
prepare a new forecast 3 7%
it depends on the timing 12 27%
Responses 44

Q2 What is the highest frequency of data at which you model/forecast?

If this varies systematically across forecast variables and/or horizons please elaborate.
Monthly 26 59%
Quarterly 11 25%
Annual basis 0 0%
Depends on the variable/horizon 13 30%
Responses 44
Note: A number of respondents indicated more than one category.

1 Any questions or queries on the questionnaire and results should be addressed to Aidan Meyler or Ieva Rubene at ecb-spf@ecb.europa.eu.

ECB
Results of a special SPF questionnaire
12 April 2009
Q3 To what extent (percentage) are your forecasts model-based or judgement-based?
If this varies systematically across forecast variables and/or horizons please elaborate
below. We recognise that this can only be an approximation and the extent may vary over
time; nonetheless, it is of interest to know how forecasts are generated. Note: Time series
models include ARIMA, single equation, VAR and factor models.

HICP inflation Short term Medium term Long term Average

Macro traditional 18% 28% 26% 24%
Macro DSGE 1% 1% 1% 1%
Time series 45% 32% 17% 31%
Judgment 37% 39% 54% 43%
Other, please specify 0% 0% 1% 0%
100% 100% 100%
Responses 41 40 42

GDP growth Short term Medium term Long term Average

Macro traditional 23% 34% 34% 30%
Macro DSGE 0% 1% 4% 2%
Time series 34% 22% 13% 23%
Judgment 43% 42% 46% 44%
Other, please specify 0% 1% 3% 1%
100% 100% 100%
Responses 42 42 41

Unemployment rate Short term Medium term Long term Average

Macro traditional 23% 31% 34% 29%
Macro DSGE 1% 1% 4% 2%
Time series 29% 22% 12% 21%
Judgment 47% 46% 49% 47%
Other, please specify 0% 0% 1% 0%
100% 100% 100%
Responses 35 36 36

Note: A number of forecasters did not provide relative weights for the models and
judgment applied, or the weights did not add up to 100%. Their answers were excluded
from the calculations. The response rate comprises the number of replies used for
calculations.

ECB
Results of a special SPF questionnaire
April 2009 13
 Q4 What types of models do you use and what weights do you attach to them?
E.g. single equation, macro model, VAR, DSGE, etc. If this varies systematically across
forecast variables and/or horizons please elaborate. We recognise that this can only be an
approximation.

Models used Relative weights reported

Time series ARIMA 22 59% 19%
Time series single equation 24 65% 24%
Time series VAR/VEC 20 54% 13%
Time series other
(e.g. factor models) 6 16% 2%
Macro-economic DSGE 2 5% 1%
Macro-economic traditional or other 26 70% 37%
Other, please specify 2 5% 3%
100%
Responses 37 25

Note: A number of forecasters provided relative weights that did not add up to 100% or
only indicated the types of models used without assigning weights. These replies were
included in the models used calculations. Calculations concerning the relative weights
attached only take into account answers with weights that added up to 100%, therefore,
the response rate reported is lower.

ECB
Results of a special SPF questionnaire
14 April 2009
Q5 If you provide information on the external assumptions (i.e. oil prices, exchange
rates, interest rates and wage growth), how do you form such assumptions?

Oil prices
In-house forecast 32 78%
Futures/market prices 13 32%
Consensus/average forecasts 3 7%
Other please explain 11 27%
Responses 41
Exchange rates
In-house forecast 36 88%
Futures/market prices 2 5%
Consensus/average forecasts 4 10%
Other please explain 4 10%
Responses 41
Interest rates
In-house forecast 41 93%
Futures/market prices 2 5%
Consensus/average forecasts 2 5%
Other please explain 1 2%
Responses 44
Wage growth
In-house forecast 35 95%
Consensus/average forecasts 3 8%
Other please explain 2 5%
Responses 37

Note: A number of respondents indicated more than one category.

ECB
Results of a special SPF questionnaire
April 2009 15
 Q6 Does your reported point forecast refer to the mean, mode or median of your reported
probability distribution in the SPF?

Mean 21 75%
Mode 2 7%
Median 5 18%
28
Do not estimate probability distributions 18
Responses 43

Note: Three respondents indicated both mean and that they do not estimate
probability distributions.

Q7 How do you calculate your reported probability distributions?

If this varies systematically across forecast variables and horizons please elaborate.

Derived from a model 7 17%

Based on a functional form 2 5%
Judgement-based 33 79%
Responses 42

Q8 How do you interpret the interval of 1.5%-1.9%?

In the ECB SPF respondents are asked to assign probabilities associated with an outcome
occurring within given ranges (e.g. 1.5%-1.9%, 2.0%-2.4%, etc.). These ranges are open
to interpretation which sometimes makes it difficult to assess the balance of risks to the
point forecast.

1.40%-1.90% 3 7%
1.45%-1.95% 31 76%
1.50%-2.00% 7 17%
Responses 41

Q9 Do you publish externally the forecasts you send to us?

Yes 28 65%
No 15 35%
Responses 43

European Central Bank, 2009

Address: Kaiserstrasse 29, 60311 Frankfurt am Main, Germany
Postal address: Postfach 16 03 19, 60066 Frankfurt am Main, Germany
Telephone: +49 69 1344 0; Website: http://www.ecb.europa.eu; Fax: +49 69 1344 6000
All rights reserved. Reproduction for educational and non-commercial purposes is permitted provided that the source is
acknowledged.
ISBN 978-92-899-0440-7 (online)
ECB
Results of a special SPF questionnaire
16 April 2009