Transportation Planning -Policy -Research -Practice

Managing the insolvable limitations of costbenefit analysis: results of an interview based study Niek Mouter, Jan Anne Annema & Bert van Wee Transportation Planning - Policy - Research - Practice ISSN 0049-4488 Transportation DOI 10.1007/s11116-014-9539-0 1 23 Your article is protected by copyright and all rights are held exclusively by Springer Science +Business Media New York. This e-offprint is for personal use only and shall not be selfarchived in electronic repositories. If you wish to self-archive your article, please use the accepted manuscript version for posting on your own website. You may further deposit the accepted manuscript version in any repository, provided it is only made publicly available 12 months after official publication or later and provided acknowledgement is given to the original source of publication and a link is inserted to the published article on Springer's website. The link must be accompanied by the following text: "The final publication is available at link.springer.com”. 1 23 Author's personal copy Transportation DOI 10.1007/s11116-014-9539-0 Managing the insolvable limitations of cost-benefit analysis: results of an interview based study Niek Mouter • Jan Anne Annema • Bert van Wee Ó Springer Science+Business Media New York 2014 Abstract Cost-benefit analysis (CBA) has been an important tool for transport planners for several decades. Despite its popularity it has often been criticized for several reasons, amongst other things, because the tool has some insolvable limitations when it is applied in practice. In this paper we examine and scrutinize the perceptions of 86 key actors in the Dutch appraisal practice for spatial-infrastructure projects with regard to three insolvable CBA limitations: (1) CBA is always incomplete; (2) Effect estimations are always uncertain; (3) Effects that are difficult to estimate have a relatively weak position. We conclude that Dutch key actors were not only able to point out these three CBA limitations and the bad management of these limitations, but they were also able to propose several (pragmatic) solutions to improve the management of CBA limitations. This paper discusses how the proposed solutions relate to solutions addressed in the literature. Moreover, we provide recommendations for further research and discuss policy recommendations that transpired from the results. Keywords Cost-benefit analysis (CBA)
Transport appraisal
Transport planning
Decision-making process Introduction Cost-benefit analysis (CBA) has been an important tool for transport planners for several decades, in particular for evaluating and ranking transport infrastructure investments (e.g. Eliasson and Lundberg 2012; Grant Muller et al. 2001; Hayashi and Morisugi 2000; Odgaard et al. 2005). Despite its popularity CBA has often been criticized for several reasons, most of N. Mouter (&)
J. A. Annema
B. van Wee TU Delft’s Transport and Logistics Group, Delft, The Netherlands e-mail: N.Mouter@tudelft.nl B. van Wee TRAIL Research School, Delft University of Technology’s Transport and Logistics Group, Delft, The Netherlands 123 Author's personal copy Transportation them related to the insolvable limitations when it is applied in practice. For instance, estimations of future project effects are inherently very uncertain (e.g. Flyvbjerg et al. 2003, 2005; Naess 2006; Naess and Strand 2012; Salling and Banister 2009). Moreover, there is an extensive body of literature (e.g. Ackerman and Heinzerling 2004; Atkinson and Mourato 2008; Boadway 2006; Kelman 2002; Martens 2011; Sen 2000; van Wee 2012) that states that the focus of CBA (and the moral view on which the method is based: utilitarianism) on the benefits and the costs that accrue from the project is insufficient as a moral view. Some projects whose costs exceed their benefits may be morally right and, contrarily, some decisions where the benefits are greater than the costs may be morally wrong. For instance, Atkinson and Mourato (2008, p. 328) state that: ‘given that equity and justice concerns often dominate discourse about social decisions, it has often struck critics as bordering on the perverse that CBA has chosen to focus its attention so squarely on efficiency’. In a previous phase of our research program, we found that a very large majority of key actors in the Dutch appraisal practice for spatial-infrastructure projects participating in our study thinks that—in spite of the insolvable limitations of the method—CBA should be used in the decision-making process for spatial-infrastructure projects (Mouter et al. 2013b).1 In our research, as well as holding in-depth, face-to-face interviews with 86 key actors, we also asked them to fill out a written questionnaire. 74 key actors responded and 73 made clear that in their view CBA should play a role in the appraisal process for spatial-infrastructure projects. In this paper we examine and scrutinize the perceptions of key actors in the Dutch appraisal practice for spatial-infrastructure projects (hereafter referred to as ‘‘the Dutch practice’’) with regard to insolvable CBA limitations, the problems these limitations lead to, and solutions to manage these problems in the decision-making process. In the in-depth interviews with the 86 key actors in the Dutch practice we pursued these three topics specifically. In our view, analyzing key actors’ perceptions of solutions that aim to manage (the problems that are a consequence of) insolvable CBA limitations is scientifically relevant in itself because, to the best of our knowledge, this focus on the perceptions of actual CBA actors regarding solutions has never been carried out before. The societal contribution of this paper is that the proposed solutions can be used to enhance the management of insolvable CBA limitations in the decision-making process. Finally, the proposed solutions can be used as inspiration for further research. The remainder of this paper is organized as follows: ‘‘Research methodology’’ section presents the research methodology. ‘‘Results: insolvable CBA limitations’’, ‘‘Results: when and why do respondents perceive that insolvable CBA limitations are problematic?’’and ‘‘Results: respondents’ perceptions of how to manage insolvable CBA limitations’’ sections present the results. Finally, ‘‘Conclusion and reflections’’ section provides concluding remarks and reflections. Research methodology The research on which this paper is based was carried out as part of a larger research program, which aspires to improve the appraisal of spatial-infrastructure projects using CBA in the Netherlands and simultaneously aims to contribute to the scientific CBA literature. In this research project the intention was to interview the entire population of 1 This paper defines spatial-infrastructure projects as both the classic infrastructure projects such as ‘highways’ and ‘railroads’ and spatial projects such as ‘integrated land use and transportation projects’ and ‘flood protection projects’. 123 Author's personal copy Transportation key actors that had an explicit and recognizable role in the Dutch practice in the last decade. To identify the key actors in the Dutch practice, we used a three-stage method (see, Mouter et al. 2013a for a more detailed elaboration of the three-stage method). In the first stage, 30 key actors were interviewed (consultants, policy makers, scientists and politicians with an important role in the Dutch practice in the last decade). We asked these 30 respondents which people, in their view, were paramount—besides the people that were already interviewed—to be certain that we interviewed the full Dutch population in this area. Based on these suggestions, in the second stage, 42 additional respondents were interviewed. In the third stage, the list of the 72 interviewed respondents was presented to four respondents that were often mentioned as key actors in the Dutch practice during the interviews. These respondents were asked to add the names of the people that needed to be interviewed to the list in order to make sure that all of the key actors in the Dutch practice were interviewed. In this third stage, 14 additional respondents were interviewed. Thus, in total, we managed to interview 86 key actors in the Dutch practice (see Appendix 1 for a list of the respondents). Fifteen other people were also approached for an interview but they were not able to, or were not interested in participating in the research. Thus, we did not manage to interview the entire population as intended and the 86 respondents must be considered as a selection of the entire population of key actors. Moreover, we think that it is possible that although we consider the three-stage method as a comprehensive way to identify the key actors in the Dutch appraisal practice for spatialinfrastructure projects it is possible that we missed a few key actors. In the interviews, amongst other things, we asked the respondents to mention the most important advantages, disadvantages, problems and limitations they experience with the CBA methodology and when using CBA in the decision-making process. Moreover, we asked them to mention the solutions they perceive for minimizing or managing the limitations and problems they identified. We used content analysis to analyze the interviews. Content analysis is defined as a systematic, replicable technique for compressing many words of text into fewer content categories based on explicit rules of coding and categorizing (Weber 1990). We coded the respondents’ quotes as a perception of an insolvable CBA limitation when the respondent stated that he/she perceives the limitation as insolvable.2 We also coded the respondents’ perceptions of problems that are the result of the insolvable CBA limitations and their perceptions of solutions to manage these problems. The content analysis resulted in 79 perceptions of insolvable CBA limitations, 95 perceptions of related problems and 159 perceptions of solutions to manage the problems that are a result of CBA limitations. We use the Netherlands as a case study because we are Dutch researchers and we know the Dutch community and procedures better than those of other countries. We think that the Dutch practice can give useful international insights for other CBA practices because of the extensive use of CBA in the Netherlands over the last 13 years. Mackie and Worsley (2013) state that, along with the UK and Scandinavia, the Netherlands has been a leading country in the international CBA practice for four reasons: (1) it has a strong tradition of doing transport project appraisal; (2) it has guidance manuals which constitute a clearly defined framework for appraisal which is to be followed throughout the project cycle; (3) it 2 Besides insolvable CBA limitations, respondents addressed the fact that a limitation of the CBA is that it does not take other elements in the decision-making process into account, other than project effects and efficiency, such as public support and political support issues. We did not consider these issues in this study because we think they are not related to ex-ante evaluation of project effects but to the decision-making process in general. 123 Author's personal copy Transportation has a framework populated with measures and values of the impacts, which are based on evidence generated from research studies (4) the Netherlands has a policy whereby the intention is that the results of appraisal work should have a significant influence on the case for investment and on prioritization within programs. Since 2000, around 110 CBAs for candidate transport policies became publicly available on a very broad range of topics (such as new roads, new railroads, tunnels, sea port extensions, airport extensions, pricing policies, speed policies, and so forth). Moreover, the projects are both on a national as well as a regional level (so, CBAs are made for projects ranging from large, national, multibillion euro projects to relatively small, multi-million, local projects). Important effects considered in the Netherlands practice are: direct transport benefits (such as travel time savings and reliability effects), wider economic impacts, and all kinds of external impacts such as environmental and safety impacts (see Mackie and Worsley 2013 for a more indepth discussion of effects that are considered in Dutch CBAs). The official Dutch CBA Guideline (OEI-Manual, Eijgenraam et al. 2000) states that CBA practitioners should aim to monetize effects as much as possible. In addition, the guideline states that specific effects (such as effects on unique landscapes) cannot be monetized in an objective way and should be expressed in the CBA report in a quantitative or qualitative way. In this paper we discuss the 132 perceptions of solutions for managing insolvable CBA limitations which we consider to be internationally relevant. In a Dutch language report (Mouter et al. 2012) we elaborate on all of the solutions mentioned in the interviews by key actors. In the next sections we do not present the number of respondents that mentioned a particular solution because one of our main aims is to generate interesting solutions for managing insolvable CBA limitations. In our view ‘interesting’ does not depend on the number of times a solution is given. Just one person can have the ‘brilliant’ solution. Nevertheless, to get an impression of the number of respondents that mentioned a solution we provide an overview of these frequencies in see Table 2 in Appendix 2 Results: insolvable CBA limitations Our results show that key actors perceive three main insolvable CBA limitations (Fig. 1): incompleteness, uncertainty and the relatively weak position of effects that are difficult to estimate. This section discusses how these three categories can be decomposed (based on the content analysis of the interviews) and describes how the respondents specifically define these limitations. We also give briefly the views on these specifically defined limitations found in international literature (if available). CBA studies are always incomplete One insolvable CBA limitation that respondents mention is that CBAs are always incomplete, because in their view it is inevitable that relevant welfare effects are overlooked in CBA studies or that it is not possible to include some welfare effects in a CBA (sometimes even not in a qualitative way) because it is not possible to estimate these welfare effects with sufficient reliability. The respondents label these types of effects as intangible effects (Fig. 1). Respondents define intangible effects as (a) effects for which it is unknown whether the effect will accrue at all; (b) effects for which it is difficult to determine the causality between the project and a claimed effect, and/or; (c) effects for which it is difficult to determine the extent to which the effect is additional or nonadditional to the national welfare. Most respondents labeled an effect as ‘intangible’ when 123 Author's personal copy Transportation Fig. 1 Insolvable CBA limitations perceived by respondents it complies with only one of the three conditions. An example of an effect that complies with the three conditions is the often claimed effect of an ‘enhanced business climate’ as a result of a spatial-infrastructure project. Other intangible effects mentioned by respondents are: effects on the image of the country, positive knowledge spillovers, effects on ‘regional identity’, agglomeration effects, positive effects of innovative projects and effects on the quality of life. Respondents’ perceptions are confirmed in scientific literature. For instance, Ambrasaite et al. (2011), Odgaard et al. (2005) and Mackie (2010) conclude that the ideal of CBA studies to take into account all the welfare effects in the study is unrealistic because it is not possible to estimate some welfare effects in a sufficiently reliable way. Börjesson et al. (2013, p. 3) state that: ‘it is uncontroversial that investment CBAs do not capture all possible relevant effects or considerations. For example, some costs and benefits cannot be accurately valued or measured.’ Another incompleteness issue mentioned by respondents is that CBA studies exclude project effects that have no effect on the welfare of a country3 but should still be evaluated in the ex-ante evaluation process, in their opinion (Fig. 1). Respondents discuss three examples of such effects. Firstly, in CBA, poor people count less than rich people because poor people’s willingness to pay is, generally, relatively low. Secondly, they perceive that the CBA is incomplete as a result of the anthropocentric perspective of the method. The CBA determines negative effects of biodiversity by the value humans assign to the loss of biodiversity as a result of a project. Biodiversity in itself has no value in the CBA. One respondent states that the CBA is not sustained on the principle that one needs to give something back to the planet when one takes something from it and perceives it as a limitation of the CBA that the method does not consider this approach at all. Thirdly, respondents perceive that the perspective of CBAs applied in practice is incomplete because CBAs predominantly tend to scrutinize the costs and the benefits of a project for a specific country. These respondents perceive that it is incorrect when CBAs in the Dutch practice only include welfare effects for the Netherlands and therefore communicate a positive message when the project results in a net positive welfare effect for the Netherlands although the project has very negative effects for other countries. 3 The OEI-Leidraad (official Dutch CBA Guideline) defines welfare effects of a project as: ‘all financial and non-financial effects of a project for Dutch residents’. Project effects on safety and the environment are used in the guideline as illustrations of non-financial welfare effects. 123 Author's personal copy Transportation In conclusion, Dutch CBAs evaluate the effects of a project on the welfare of the country; however the evaluation of specific types of distribution of welfare (between rich versus poor people, humans versus nature and Dutch residents versus non-Dutch residents) is often neglected both in Dutch CBA studies and the wider ex-ante evaluation process of spatial-infrastructure projects. Martens (2011), Rietveld et al. (2007), Thomopoulos et al. (2009) and van Wee (2012), amongst others, also discuss that the distributional effects of a project tend to be underexposed in CBA studies. Outcomes of welfare effect estimations are always uncertain Respondents also mention—as an insolvable limitation of the CBA—that estimations of welfare effects of a project in a CBA are very uncertain (Fig. 1, middle box). The first main reason mentioned is that consultants who carry out ex-ante CBAs need to make assumptions for the unknown future when they want to estimate effects. Secondly, in practice, consultants do not have the time and money to estimate all the effects of a specific project from scratch, and as a result they have to use shortcuts, national standard numbers and rules of thumb to estimate the effects of a specific project, which leads to uncertainty. The inherent uncertainty of ex-ante effect estimations in CBA studies is also abundantly addressed in scientific literature (e.g. Beattie 1995; Flyvbjerg et al. 2003; Nicolaisen 2012; Self 1970). Effects that are difficult to quantify/monetize have a relatively weak position Some respondents state that an insolvable limitation of CBA is that effects that are difficult to quantify/monetize—such as effects on biodiversity—per definition, have a relatively weak position in CBA reports4 (Fig. 1, right) compared to impacts that are easy to quantify/monetize, such as construction costs and transport benefits. Because project effects on biodiversity are difficult to quantify/monetize, consultants that carry out CBAs frequently decide to include them in a non-monetized way, which gives these welfare effects—according to some respondents—a relatively weak position in the CBA report compared to monetized effects. Respondents state that this relatively weak position cannot be rectified by monetizing the effects, for two reasons. Firstly, they state that monetization leads to the risk that—as a result of quantification and monetization problems—only a part of the effects on biodiversity are monetized and taken into account in the CBA report, whilst the reader thinks that the total effect is included. Secondly, some respondents oppose the monetization of effects, like effects on biodiversity and traffic casualties, because they think it is incorrect to transfer these types of effects into the same unit as effects like travel time savings and construction costs.5 Existing literature (e.g. Mackie and Preston 1998; Mishan 1988) also mentions this limitation. Mishan (1988) addresses this insolvable limitation as follows: ‘if you take one horse and one rabbit, no matter how you combine them the taste of horse dominates the 4 There is no clear consensus in the Dutch practice that this is an insolvable CBA limitation. Some respondents perceive that this is an insolvable limitation; others perceive that it is currently-or will in the near future be - possible to quantify and monetize all effects that are included in a CBA study equally well. For example, with a more frequent use of insights obtained from experimental economics and behavioral economics. 5 This position of respondents is more extensively discussed in Sect. 5. 123 Author's personal copy Transportation stew. Similarly, if you take one set of quantifiable impacts and one set of non-quantifiable impacts in an appraisal, one set will dominate’. Results: when and why do respondents perceive that insolvable CBA limitations are problematic? In the interviews, respondents frequently mentioned that the fact that CBA is a tool with insolvable limitations is not problematic in itself. According to the respondents, problems only arise when insolvable limitations are not managed properly. Respondents perceive that, as a result of bad management, actors might use or position the CBA as an instrument with too many or too few limitations and attribute an incorrect value to the CBA (either too much or too little value). Below, we will discuss this respondents’ position in more depth and, again, we also give views taken from existing literature on the positions discussed. Why are insolvable limitations problematic? Figure 2 displays the lines of reasoning used by the respondents to explain why they think that the insolvable CBA limitations can be problematic in this respect. The respondents’ arguments frequently had different steps to them. The different steps are depicted by the arrows and characters in Fig. 2. Fig. 2 Problems resulting from the perceived insolvable limitations according to respondents 123 Author's personal copy Transportation More detail will now be given about the two most important observations that can be derived from Fig. 2. Imperfect presentation of CBA limitations leads to assigning incorrect value to CBA The first observation is that respondents perceive that—as a result of imperfect presentation of CBA limitations in the CBA report—readers find it very difficult to understand the limitations of the CBA sufficiently. One respondent states: ‘how is it possible for a politician to know that assumptions are very contestable when the CBA report does not communicate this explicitly in the executive summary of the report?’ Respondents observe that the chance that users understand the CBA limitations in an insufficient way as a result of an imperfect communication of CBA limitations in the CBA report is especially high when users have limited time to read the report. Here, respondents emphasize that especially politicians—who are important users of CBA studies—have limited time to read a CBA report and frequently only read the summary of a CBA report, whilst summaries especially tend to neglect the communication of limitations. Figure 2 shows that respondents perceive that, as a result of the imperfect presentation of CBA limitations, some actors might assign too much value to the CBA (a–b in Fig. 2), but others might assign too little value (a–c in Fig. 2). They state that some actors in the Dutch practice (unintentionally) assign too much value to the CBA because they are not aware of CBA limitations and, as a result, use it as a holy grail (‘‘we decide positively only if the benefit-cost ratio is above 100 ). On the other hand respondents state that a lack of communication of CBA limitations enhances suspicion by skeptical actors, which leads to a situation where these actors assign too little value to the CBA (‘‘I don’t trust instruments that produce false certainties and are not honest about the limitations of the results’’). The fact that CBA reports tend to communicate limitations of the study insufficiently is endorsed in the literature. According to Welsh and Williams (1997), CBA outcomes are usually presented as if they are endowed with considerable accuracy, even though estimations of traffic models, for instance, are very uncertain. Naess and Strand (2012) state that the high degrees of uncertainty are often not displayed in cost-benefit analyses. From an analysis of decision support documents for 78 Norwegian and Danish road projects, Nicolaisen (2012, p. 7) finds that: ‘‘uncertainties are often toned down or ignored in the decision support prepared for policy makers. This neglect makes impact appraisals appear more accurate than warranted, which causes distrust towards the results among policy makers’’. Out of an evaluation of the Dutch practice in 2002 (BCI, 2002) it could be seen that, although effect estimations in CBA studies are very uncertain, CBA reports present the outcomes of CBA studies as exact truths. Strategic use of CBA limitations leads to assigning incorrect value to CBA The second main observation related to Fig. 2 is that respondents perceive that CBA limitations are also used in a strategic way by actors in the Dutch practice. To illustrate this, a respondent states that: ‘‘if the outcome of the CBA does not support the political interest of an actor, the actor emphasizes the limitations of the method, and when the outcome of the CBA supports the political interest of an actor, the actor ignores its limitations.’’ Hence, respondents perceive that as a result of strategic use of CBA limitations, some actors might assign too much value to the CBA by neglecting the limitations (d–e in Fig. 2), but others assign too little value to the CBA by overemphasizing the limitations (d–f in Fig. 2). 123 Author's personal copy Transportation In addition to the respondents’ perceptions, de Jong and Geerlings (2003) discuss that strategic behavior is interrelated. They conclude that when people assign absolute value to the CBA in a discussion, other people in that discussion—who are more skeptical about CBA—will be prone to marginalize the value of CBA as a reaction. Why is assigning an incorrect value problematic? In ‘‘Why are insolvable limitations problematic?’’ section we discussed the different causes respondents perceive for assigning an incorrect value to the CBA in the decision-making process. Now, we discuss why respondents perceive attributing an incorrect value (too much or too little, Fig. 2) to CBA as being problematic? They mention several reasons. Firstly, they state that the quality of discussions among key actors in decision-making processes on the usefulness, necessity and design of a new spatial-infrastructure project could deteriorate. Respondents state that the contemplation, discussion and decisionmaking on the usefulness, necessity and design of a spatial-infrastructure project are not enhanced when frustrations about the perceived incorrect use of CBA dominate the debate. Secondly, they perceive it as undesirable when (other) key actors attribute marginal value to the CBA in the decision-making process for spatial-infrastructure projects because attributing a marginal value to CBA will lead to a marginal utilization of the advantages of the use of CBA in the decision-making process.6 A third and final reason mentioned is that using the CBA as a ‘holy grail’ (too much value, i.e. when the CBA outcome is positive, a project should be developed; when the CBA outcome is negative, a project should not be developed) leads to four different problems. A first problem related to using CBA as a ‘holy grail’ perceived by respondents is that effects which are not or are not taken well into account in a CBA, are erroneously neglected or have a relatively weak position in the decision-making process (‘‘if you assign absolute value to the CBA in the decision-making process, you neglect other effects that are important in the decision-making process and then your ally becomes your enemy’’). Secondly, respondents state that experts know that there is hardly any difference between a benefit-cost ratio of 1.1 and a benefit-cost ratio of 0.9. However, in practice, respondents find that the former project is marked as a fantastic project and the latter project is considered as a bad project, which they perceive as problematic. Thirdly, they state that using CBA as a ‘holy grail’ catalyzes manipulation in regard to effect estimations in CBA studies (‘‘carrying out a CBA is useless when the initiator of a project only receives financial resources when the CBA score is positive, so the initiator of the project adjusts the effect estimations until the CBA score is positive’’). Fourthly, attributing absolute value to the CBA is seen as problematic because respondents perceive that assigning absolute value to a tool with major limitations will eventually lead to the collapse of the instrument in the long run (one respondent denotes this as ‘overshoot and collapse’). This final statement from the respondents is endorsed in the literature. According to De Jong and Geerlings (2003), the use of the CBA in an absolute way will lead to the resistance of other actors, which can become so strong that at a certain point CBA will lose the necessary political support. 6 Perceived advantages of key actors in the Dutch practice are discussed in Mouter et al. (2013b). 123 Author's personal copy Transportation Results: respondents’ perceptions of how to manage insolvable CBA limitations Respondents perceive that the fact that CBA is a tool with insolvable limitations is not problematic in itself. According to the respondents, problems only arise when actors use or position the CBA as an instrument with too many or too few limitations as a result of bad management of these insolvable issues. This section describes the solutions proposed by the respondents for managing the perceived problems that result from the insolvable CBA limitations. Table 1 gives an overview of these solutions and shows that some proposed solutions aspire to minimize insolvable CBA limitations, whereas other solutions aspire to manage the insolvable CBA limitations and stimulate a ‘correct’ use or ‘subtle’ use of the CBA in the decision-making process.7 Table 1 also shows that for some proposed solutions there is controversy among respondents with regard to the ‘best solution’ and for others no contradicting remarks were found in the interviews.8 Below, we first discuss the proposed solutions that aspire to minimize perceived insolvable limitations. According to the respondents, minimizing the insolvable CBA limitations could possibly downsize the problems that result from CBA limitations. Next, we discuss the proposed solutions that aspire to manage the problems that result from the insolvable CBA limitations. ‘Actor participation’ as a solution—to take more welfare effects into account (solution 1 in Table 1) One solution that respondents propose is to organize ‘effect survey meetings’ (Effect Arenas) preceding a CBA study to take into account as many welfare effects as possible in the CBA study and, as a result, minimize the limitation that CBA studies are always incomplete. They state that all relevant actors affected by the project should be invited to this meeting. The purpose of this meeting is that all actors should get the opportunity to articulate which effects they think will accrue as a result of the project. To be clear, only effects that increase national welfare are included in the CBA (the idea of the Effect Arena is that the CBA analysts can clarify to other actors why the non-welfare effects will not be included in the CBA report). Respondents perceive that, as a consequence of such a meeting, effects that would be overlooked without an Effect Arena will now be included in the CBA study. One respondent states that he experienced in practice that the efficiency of these ‘effect survey meetings’ could be optimized when the people responsible for the organization of the meeting communicate to the invited actors, in advance of the meeting, that this meeting will be the only opportunity to articulate the project effects that will be scrutinized in the CBA study. Moreover, respondents state that an additional benefit of ‘effect survey meetings’ with actors might be that public support for the use of CBA to appraise spatial-infrastructure projects will increase. 7 Mouter et al. (2012) denote the ‘subtle’ use of the CBA, based on Aristotle’s Ethica Nicomachea: ‘virtuous use of the CBA in the decision-making process for spatial-infrastructure projects’. Aristotle defines acting ‘virtuously’ as acting at the balance point between two non-virtuous extremes. Like acting ‘bravely’ is the balance point between acting ‘cowardly’ and acting ‘recklessly’, attributing a ‘virtuous’ value to CBA in a decision-making process is the balance point between assigning absolute value to the CBA and marginalizing the value of CBA. 8 We cannot say that there is consensus because we have not confronted all actors with all solutions. 123 Proposed solution to manage incorrect use of CBA Which limitation does the solution aspire to minimize or manage? Does the solution aspire to minimize or manage the CBA limitation? Is there controversy on the ‘best solution’ among respondents? 1. Organize effect survey meetings Incompleteness Minimize No contradictory remarks 2. Provide high quality qualitative information with regard to intangible effects Incompleteness Minimize No contradictory remarks 3. CBA reports should be supplemented with an analysis of distributional effects Incompleteness Minimize No contradictory remarks Uncertainty Minimize No contradictory remarks 5. Monetize as much as possible Relatively weak position of effects that are difficult to monetize Minimize Controversy 6. Use indicators other than money to express effects that are difficult to monetize Relatively weak position of effects that are difficult to monetize Minimize Controversy 7. Monetize effects in CBA studies whenever possible. However, when monetizing leads to severe criticism and frustrations one should not monetize Relatively weak position of effects that are difficult to monetize Minimize Controversy 8. Referent assesses the position of effects that are difficult to monetize in the CBA report Relatively weak position of effects that are difficult to monetize Minimize No contradictory remarks 9. Explicitly discuss the way monetized and nonmonetized effects relate to each other in the summary of the CBA report Relatively weak position of effects that are difficult to monetize Minimize No contradictory remarks 10. Implement a ‘showstopper’ procedure in the CBA analysis for effects that difficult to monetize and that are irreversible Relatively weak position of effects that are difficult to monetize Minimize No contradictory remarks 11. Enhance communication about the incompleteness in CBA reports Incompleteness Manage No contradictory remarks 12. Enhance communication about the uncertainty in CBA reports Uncertainty Manage Controversy 13. Estimate the project effects for at least one flexible investment strategy in a CBA study Uncertainty Manage Contradictory remarks 14. Communicate the fact that readers should also take into account the effects that are difficult to monetize Relatively weak position of effects that are difficult to monetize Manage No contradictory remarks Author's personal copy 123 4. Carry out ex-post analyses Transportation Table 1 Proposed solutions for managing problems that result from the insolvable CBA limitations Author's personal copy Transportation Fig. 3 Erasmus bridge in Rotterdam, example of an architectonical innovative bridge Try to provide high quality qualitative information with regard to intangible effects (solution 2 in Table 1) Respondents state that CBA analysts should not decide too quickly that it is not possible to take intangible welfare effects into account in a CBA study because it is not possible to estimate the welfare effect with sufficient reliability. They propose that analysts should strive to provide high quality objective information with regard to these effects. For example, one respondent states that if decision makers believe that a new bridge will contribute to a positive image of their region (such as the Erasmus bridge in Rotterdam, see Fig. 3), then the CBA should provide qualitative information regarding ‘the effect of architectonically innovative bridges on the image of the region’. This could be carried out by addressing what the success and failure factors of architectonical innovative bridges on the image of the region are and what the potential benefits of a successfully ‘image enhancing bridge’ are. According to the respondent, the specific CBA should discuss the extent to which the identified success and failure factors are entailed in the specific bridge under scrutiny in the CBA study. Supplement CBA reports with an analysis of distributional effects (solution 3 in Table 1) Dutch CBAs evaluate the effects of a project on the welfare of the country. However, information regarding the distribution of welfare is often underexposed or even neglected. Respondents propose minimizing the underexposure of distributional effects by including a discussion on the relevant distributional effects in the CBA report. According to these respondents, CBA studies do not have to exclude a discussion of distributional effects. The respondents state that CBA analysts should display separate ‘balance sheets’ for relevant stakeholder groups besides the national CBA results (for instance, a separate CBA for relevant regions, interest groups, income classes). The respondents emphasize that including a ‘distributional analysis’ in the CBA report ensures that the CBA study can be used by politicians because politicians are interested in the distributional effects as well as the effects 123 Author's personal copy Transportation of a project on national welfare. One respondent states that Dutch CBA guidelines should prescribe supplementary distributional analyses in a more compelling way. Carry out ‘ex-post’ analyses to diminish uncertainty with regard to effect estimations (solution 4 in Table 1) As a strategy to diminish uncertainty with regard to effect estimations, respondents propose establishing an obligation within Dutch planning guidelines to carry out ex-post analyses to verify the ex-ante estimations of project effects in the CBA. Respondents think that ‘expost analyses’ could help by improving models, rules of thumb and standard national numbers that are used to estimate project effects for specific CBA studies, which could help to reduce uncertainty by improving models. This is confirmed in the literature (e.g. Salling and Banister 2009; Nicolaisen 2012). Moreover, one respondent states that an additional benefit of institutionalizing ex-post analysis might be that the trust in government decision making would be enhanced because the government shows that it really does check the quality of the information it uses in decision-making processes. Controversy about the ‘best strategy’ for taking into account effects that are difficult to monetize in a CBA study (solutions 5, 6, 7, 8, 9 and 10 in Table 1) Respondents concede that effects that are difficult to monetize should be discussed in the CBA analysis. However, they disagree to a great extent when answering the question about how to minimize the relatively weak position of effects that are difficult to monetize in a CBA study. Respondents mentioned six different solutions. The first three solutions (5, 6 and 7) focus on the question about whether or not monetization of effects that are difficult to monetize helps to minimize the relatively weak position of these effects in CBA studies. The last three solutions discuss other solutions (7, 8 and 9). Below, we firstly discuss solutions 5, 6 and 7. In a broad sense, two of these three solutions can be labeled as extreme positions: ‘monetize as much as possible’ (solution 5) and ‘use other indicators than money to express effects that are difficult to monetize’ (solution 6), and the other one is a middle position (solution 7). Next we discuss the three positions in more depth. The first group of respondents emphasizes that it is important to monetize effects as much as possible, for two reasons. Firstly, they perceive that monetizing an effect in a CBA study is the only way to guarantee a serious position for the effect in the decisionmaking process because they perceive that there is a very high risk that decision makers will, in the end, only look at the monetary figures that are presented in the CBA report. Secondly, they state that an important benefit of the CBA is that studies provide insight into the order of magnitude of different welfare effects and the ratio of costs versus benefits of a project and that this benefit can only be utilized in an optimal way when one tries to monetize as many effects as possible. These respondents argue that new research could improve valuation methods and standard numbers that can be used to monetize effects in CBA studies. This group of respondents believes that it will eventually be possible to monetize all welfare effects equally well—for instance with a more frequent use of experimental economics and behavioral economics—which implies a rectification of the limitation that some effects are more difficult to monetize than others. The second group of respondents states that some effects (e.g. the effects on traffic casualties and the effects on biodiversity) should not be transferred into monetary terms because they are incommensurable, which implies that the nature of the effects resists 123 Author's personal copy Transportation transferring them into a common unit with other effects, such as travel time savings. According to the respondents, transferring incommensurable effects into a common unit in the CBA report erroneously communicates to decision makers that it is feasible to outweigh negative effects on biodiversity, for instance, and positive effects on travel time savings, for instance. The respondents accept that, as a result of taking into account the effects on biodiversity in non-monetary terms in CBA studies, these effects can have a relatively weak position in the decision-making process. As an alternative for monetization, this group of respondents proposes using a ‘nature value indicator’,9 which measures the effects of a project on biodiversity in weighted hectares. The weight is determined by the ecological quality of the area and the degree of threat of the ecosystem(s). The indicator proposed is called T-EQA, an acronym of Threat-weighted Ecological Quality Area. This T-EQA (Sijtsma et al. 2011; Sijtsma et al. 2013a) aims to measure the effect of a project on biodiversity from an ecological perspective (what is the effect of a project on the biodiversity in the Netherlands?) using a standardized ratio scale measurement. The ‘nature value indicator’ aims to measure project effects on biodiversity as objectively and rigorously as possible and aggregates this evaluative information in one indicator (T-EQA). Its relation to welfare economics is subtle. Instead of being from the perspective of welfare economics (what is the effect of a project on the welfare of Dutch citizens?) it uses a ‘mere’ ecological perspective. On the other hand it can be argued that biodiversity preferences concern citizens’ (non-consumer) preferences (Sijtsma 2006). Or, in Maslow terms, they concern the higher order needs (Sijtsma et al. 2013a). T-EQA then measures the size of something relevant to well-being or welfare, without giving an a priori weight or fixed monetary value per unit (Sijtsma et al. 2013a). A third group of respondents can be positioned in the middle of the two groups of respondents described above. This group perceives that initiators of CBAs and CBA analysts should base their decision about monetizing or not monetizing effects in CBA studies on pragmatic reasons. More specifically, the respondents state that one should monetize effects in CBA studies when possible. However, when initiators and consultants feel that monetizing effects will give rise to severe criticism by important actors in the decision-making process, and when this will lead to a situation in which frustrations regarding monetization of the effect will dominate the debate, one should not monetize the effect. Next, we discuss the three other solutions (8, 9 and 10 in Table 1). The first solution proposed by respondents suggests that a more equal position for effects that are difficult to monetize can be safeguarded when an independent referent assesses in a draft version of the CBA report whether the position of effects that are difficult to monetize in the report is strong enough.10 The second solution entails that the position of effects that are difficult to monetize in the decision-making process can be improved when CBA practitioners explicitly discuss the way monetized and non-monetized effects relate to each other in the summary of the CBA report. The respondent illustrates his suggestion as follows: ‘CBA practitioners should, for instance, divide the net present value (NPV) of all monetized effects by the number of hectares of nature that are sacrificed as a result of the project. Hence, one can provide the decision maker with the ‘value of a sacrificed hectare’, for instance, 50,000 euros NPV per hectare. This enhances the attention for non-monetized effects.’ The third solution suggests implementing a ‘showstopper’ procedure in the decision-making process for spatial-infrastructure projects for effects that are difficult to 9 10 Natuurpunten (in Dutch). The respondent did not discuss this solution in more depth during the interview. 123 Author's personal copy Transportation monetize and that are irreversible (for instance, the effects of the project on endangered species). According to the respondents, the CBA report should explicitly address whether the non-monetized effect exceeds an ‘environmental constraint’. When this environmental constraint11 is exceeded, the project cannot be developed in spite of a positive welfare effect. This solution is implemented in the German appraisal practice for infrastructure projects (Mackie and Worsley 2013). Enhance communication of incompleteness in the CBA report (solution 11 in Table 1) Respondents propose enhanced communication about the incompleteness of CBA studies in the CBA report as a solution to manage the inherent incompleteness of CBA reports. They emphasize the benefits of prominent communication about incompleteness in the summary of the CBA report. Respondents perceive that, as a result of articulating incompleteness in a prominent way in the CBA report, actors in the Dutch practice are less prone to attribute absolute value to the CBA in the decision-making process. Moreover, respondents perceive that the explicit communication of the elements and effects that are not considered in the CBA study will diminish the tendency of some actors to marginalize the value of CBA in the decision-making process because they feel that the CBA is honest about its limitations. Enhance communication about the uncertainty of effect estimations in the CBA report (solution 12 in Table 1) Respondents propose a variety of methods to communicate uncertainty in CBA reports (e.g. presenting bandwidths instead of point estimates, carrying out more sensitivity analyses, using more future scenarios, articulate uncertainties in an explicit and prominent way in the summary of the CBA report, for instance by discussing the assumptions that need to be made to change the sign of the outcome of the CBA12). One respondent takes it a step further and states that the most effective way of communicating uncertainty is by carrying out a large amount of sensitivity analyses on the most important assumptions in the CBA study and, subsequently, presenting the outcomes of these sensitivity analyses in a scatter plot. Moreover, respondents state that CBA analysts should—as a result of the severe uncertainties—restrict themselves to present the effect estimations and never make suggestions in regard to the possible consequences of the results for the decision-making process. One respondent states that the passage: ‘the outcome of the CBA score is negative, thus the development of the project will have a negative effect on the welfare of the Netherlands’ is already a bridge too far, given the uncertainties. Mouter et al. (2013b) conclude that another group of Dutch key actors suggests that one must be cautious about communicating CBA limitations in a way that is too prominent. Some even perceive that a too prominent communication of uncertainties will lead to ‘the collapse’ of the instrument in the decision-making process. Below, we discuss two risks that they perceive that are related to a too prominent communication of uncertainties. Firstly, respondents state that politicians prefer certain, plain and easy to comprehend information over uncertain and nuanced information. One respondent states that: 11 The respondent did not discuss precisely which constraints he had in mind. 12 One respondent suggests communicating in a specific CBA in which the effects of a new rail connection and a spatial planning project were assessed, that the BCR ratio will only be positive when one assumes that housing prices in the project area will increase by 20,000 euros as a result of the new rail connection. 123 Author's personal copy Transportation ‘politicians desire a study that reports only one figure in the conclusion, which communicates whether or not the study supports the project’. Another respondent perceives that politicians’ preference for certain information over uncertain information could be explained by a specific inclination that politicians have of making some decisions during their term of office. The respondent states that research reports which communicate uncertainty, in most cases lead to a delay in the decision-making process and that this delay can be very undesirable for politicians. Moreover, respondents perceive that politicians will not consider a CBA report that communicates an uncertain message as a solid base for decision making. According to the respondents, this might eventually lead to a situation in which the use of CBA in the decision-making process will be abandoned. Secondly, some respondents perceive that prominent communication of uncertainty in CBA reports will lead to more strategic use of the CBA in the decision-making process, which they feel would be undesirable. Estimate the effects of at least one flexible investment strategy in a CBA (solution 13 in Table 1) A group of respondents thinks that one should manage uncertainty regarding estimations of project effects by estimating project effects for at least one flexible project alternative alongside the non-flexible project alternatives.13 For instance, in a CBA effects are estimated for a new railroad which connects cities A, B, C and D (non-flexible project alternative). Here, the idea is to include a project alternative in this CBA in which a new railroad connects cities A and B and additional busses connect cities B, C and D. When the railroad between A and B is a success, the line can be extended to cities B, C and D (this is the flexible project alternative). Hence, for the flexible project alternative it is assumed that the project is planned in an incremental way. The group of respondents is in favor of this idea because only the elements of the project are developed for which it is to some extent certain that the benefits will very likely be higher than the cost. Respondents label these elements as ‘no regret’ elements. If the future develops in a favorable way for the project, the other elements can be built as well. When there is an unfavorable development, the other elements are not developed or alternative elements are developed. With real option analysis it is possible to assess the costs and benefits of a flexible investment strategy against the strategy to develop the whole project in one go (see, for instance, Reuer and Tong, 2007 for a discussion regarding the merits of this method). Respondents perceive as a potential benefit of a flexible strategy the fact that one can adapt the design of the project to the way the future unfolds, especially when the future unfolds in a different way than expected. They perceive as a potential downside of such a strategy the fact that residents will live for years in uncertainty as to whether the second part of the project will or will not be built. According to respondents, a flexible strategy might also be disadvantageous when the specific transport market is characterized by a so-called ‘first mover advantage’. For example, when newly developed large container vessels can only call at one Northern European harbor, it is very likely that this harbor will capture all the benefits of large container vessels. In that case, an incremental planning process where the expansion of a harbor starts after the large container vessels are on the market might have its downsides. Respondents perceive that real option analysis can assist with outweighing 13 For some types of projects it might be very difficult to design a flexible investment strategy (for instance, bridges). For other types of projects it might be easier to develop flexible investment strategies (for instance, airport and harbor extensions). 123 Author's personal copy Transportation the benefits and the costs of a flexible investment strategy compared to a non-flexible investment strategy. Some respondents also discuss the downsides of using real option analysis for evaluating flexible investment strategies. These respondents doubt whether it is possible for decision makers and other users of CBA reports to understand the results of the analysis. Moreover, for the application of real option analysis it is necessary to determine the probability that the different futures included in the analysis will unfold. Respondents think that it is problematic to convince users of CBA reports that it is possible to determine these probabilities. Communicate non-monetized effects in a prominent way (solution 14 in Table 1) Respondents propose that the summary of the CBA report should communicate to the readers of the CBA report in a very prominent way that they should take into account both monetized and non-monetized effects. Respondents emphasize that it is important to manage the way non-monetized effects are taken into account in the CBA report in order to make sure that their position in the decision-making process is as equal as possible compared to monetized effects because they perceive that actors in the Dutch practice use this lack of balance as the main argument to oppose the use of CBA. Conclusion and reflections Although we did not manage to interview the entire population of key actors in the Dutch appraisal practice for spatial-infrastructure projects—fifteen people approached were not able to or were not interested in participating in this research—we are confident that it is possible to make a general conclusion by stating that most key actors in the Dutch practice acknowledge insolvable CBA limitations, but perceive the CBA—in spite of these limitations—as a useful instrument.14 A large proportion of the respondents think that insolvable CBA limitations can be managed and were able to come up with several pragmatic solutions. We think this is an interesting conclusion because it contrasts with the conclusions of some publications (e.g. Naess 2006; Kelman 2002) that take the insolvable CBA limitations as an argument to oppose the use of CBA in the decision-making process at all. We can also conclude that the key actors perceive that if the insolvable CBA limitations are not managed properly, decision makers might attribute an incorrect value to the CBA results (either too much or too little value). Furthermore, we can conclude that, although some of the proposed solutions have already been addressed in scientific literature, several of the proposed solutions, to our knowledge, add to the literature (solutions 1, 2, 8, 9, 11, 14 in Table 1), and may inspire other CBA practices in the world to make improvements. Naturally, some proposed solutions might already be implemented in other countries. It is out of the scope of this study to scrutinize solutions to manage insolvable CBA limitations in other countries in an extensive way and relate these to the Dutch practice. However, this study could be a building block for ‘the superior international CBA model’, suggested by Hayashi and Morisugi (2000). They state (p. 87) that ‘by conducting a careful study on the components of the different models, it would be possible to come up with a superior model 14 In Mouter et al. (2013b) we discuss empirical results on which we sustain our claim that Dutch key actors perceive CBA as being a useful instrument in the decision-making process. 123 Author's personal copy Transportation by integrating all the good components of the existing models. This is a simple case of learning from each country’s experience’. A suggestion for generating more building blocks is to replicate this research in other CBA practices and review the solutions implemented in other countries to manage insolvable CBA limitations. The remainder of this section discusses the specific proposed solutions. We discuss, amongst other things, how the proposed solutions relate to solutions addressed in the literature. Moreover, we provide recommendations for further research and discuss policy recommendations that ensue from the results. Key actors in the Dutch practice agree that ‘incompleteness’ is an insolvable CBA limitation. Whether incompleteness is truly insolvable is debatable because it is conceivable that knowledge developments in the more distant future will make it possible to take into account (far) more of the welfare effects of a spatial-infrastructure project compared to the current practice. However, nobody can foresee when and whether it will be possible to measure project effects on ‘regional identity’, for instance, with sufficient reliability in the distant future. Therefore, we think that the seemingly non-controversial solutions proposed for managing the incompleteness of welfare effects (such as: organizing effect survey meetings; providing high quality qualitative information about intangible effects) are interesting extensions to the existing literature. Nevertheless, for the proposed solution to ‘enhance communication of incompleteness in CBA reports’, we recommend further research about the way incompleteness should be communicated in CBA reports in advance of an implementation decision, because—even though the key actors interviewed did not mention this—there might be a possible hazard that a too prominent communication of incompleteness would lead to ‘the collapse’ of the CBA—as would a too prominent communication of uncertainties. We believe that the proposed ‘Effect Arena’ solution can potentially add value to CBA practices. This solution will not necessarily lead to a situation where all the welfare effects of a project are included in the CBA study. However, the solution safeguards that all the welfare effects perceived by important stakeholders are discussed in the CBA study (in the CBA report, besides the effects that increase national welfare, CBA practitioners also discuss why effects which are perceived as welfare effects by stakeholders do not add to the welfare and, as a result, are excluded from the CBA score). An important benefit of conducting ‘Effect Arenas’ is that the probability that on-going (and often unclear) debates about a CBA study being complete or incomplete might diminish. We think that this solution is transferable to other CBA practices as well because the incompleteness issue is raised in other countries that use CBA (see for instance, Barfod et al. 2011; Eliasson and Lundberg 2012). Effect Arenas can help with reducing this issue. Related to the incompleteness issue, it indeed seems very important to discuss distributional effects in the CBA report, as the respondents remarked. Respondents suggest displaying separate ‘balance sheets’ for relevant stakeholders. In the literature we find guiding principles, which in essence resemble this solution as proposed by Dutch actors. Nyborg (2012) and the review of the Norwegian CBA practice (Hagen 2012) state that distributional consequences may be summarized as a list of winners and losers, supplementing the findings from a CBA. The goal of a distributional analysis is that decision makers should receive information about which conflicts of interest the project gives rise to, thus enabling them to evaluate for themselves how to address the resulting trade-offs. Martens (2011) argues that the substance of the distributional analysis should depend on the distributional concerns of decision makers and/or the wider public in a specific case. He states that, firstly, the distributive concerns in a case should be specified before developing an adequate methodology that can address these distributional concerns. In conclusion, the 123 Author's personal copy Transportation guiding principles mentioned by the respondents and stated in the literature for discussing distributional effects within or alongside the CBA report are essentially the same; however we observe minor differences. Respondents predominantly mentioned a specific distributive concern that should be addressed in CBA reports, such as distributions over regions or income classes, whereas Martens (2011) states that CBA practitioners should firstly scrutinize which distributive concerns are relevant in a specific case. We recommend future research, studying which specification works best in practice. Which solution rectifies the underexposure of distributional effects in a CBA study in the most efficient way? Besides the solutions to incorporate distributional effects in ex-ante evaluation mentioned by the respondents, we also advise including other solutions proposed by the scientific literature and other CBA practices in this study, such as ‘Incorporating distributional effects in a CBA by distributive weights’ (e.g. Boadway 2006; Campbell and Brown 2003; Mishan 1976), ‘Incorporating distributional effects in a hybrid model which combines CBA and Multi-criteria analysis’ (e.g. Thomopoulos et al. 2009; Thomopoulos and GrantMuller 2013; van Wee, 2012) and the UK Department for Transport’s WebTAG guidance on Social and Distributional Impacts (see www.dft.gov.uk/webtag/). Subsequent to this study, we recommend considering the solution that minimizes the underexposure of distributional effects in the most efficient way in the Dutch practice, but also in other practices. Many respondents stated that institutionalizing systematic ‘ex-post’ analyses in the Dutch planning and appraisal process for spatial-infrastructure projects might be an effective solution to minimize uncertainty in CBAs. Internationally, we see that in some countries ‘ex-post’ analyses are incorporated in the process (e.g. POPE Guidelines in the United Kingdom), whereas in other countries this is not the case (such as the Netherlands). We think it is interesting to study exactly how ‘ex-post’ analysis is institutionalized in different countries, and more importantly, what are the most important success and failure factors for institutionalizing systematic ‘ex-post’ analyses in the planning and decisionmaking process? Practitioners who aspire to institutionalize ‘ex-post’ analyses in their practice could use the results of this study. Key actors in the Dutch appraisal practice for spatial-infrastructure projects agree that ‘uncertainty’ is an insolvable CBA limitation. However, there is controversy among respondents both in regard to ‘the best solution’ to enhance the communication of uncertainty in CBA reports and regarding the graduation in which uncertainty should be emphasized in CBA reports. We recommend further research that implies experimentation with the different solutions proposed by respondents and different graduations in which uncertainty is emphasized. We recommend including the solution ‘display the uncertainty of effect estimations with Monte Carlo Analysis and interval results’ in this experiment. This solution was not mentioned by Dutch respondents but is broadly discussed in recent literature (e.g. Ambrasaite et al. 2011; Barfod et al. 2011; Salling and Banister 2009). From de Jong et al. (2007) we derive that it is already possible to scrutinize the uncertainty in traffic forecasts through Monte Carlo Analysis for the Dutch transport model, which is used to estimate a project’s transport effects in a CBA. Mouter et al. (2013c) discuss that the controversy between Dutch key actors regarding the graduation in which uncertainty should be emphasized in the CBA report can be explained by the way their ‘cognitive styles’ diverge. Cognitive styles are defined as individuals’ chronic motivations that principally determine the initiation, course, and cessation of information seeking and processing (e.g. Thompson et al. 2001). Mouter et al. (2013c), amongst other things, discuss that individuals who are prone to process information in a ‘systematic’ way are more prone to evaluate information that communicates an uncertain message as a useful 123 Author's personal copy Transportation input in the decision-making process than individuals who process information in a ‘heuristic’ way. We recommend a study of relevant findings from social psychological scientific literature in research programs that focus on an optimal way to communicate uncertainty in CBA reports. The Dutch spatial-infrastructure planning practice focusses more and more on flexible investment strategies. We recommend assessing the advantages and disadvantages of scrutinizing both flexible investment strategies and non-flexible investment strategies in real-world CBAs through real option analysis in the near future. It would be interesting to monitor people’s experiences with this method. Is this an effective and efficient solution to manage the insolvable limitation that project effect estimations in a CBA are uncertain? When the experiences are positive this solution could be promising for other countries as well. Our empirical research shows that there is controversy in the Dutch practice regarding the insolvability of the relatively weak position of effects that are difficult to estimate and monetize. Respondents propose different solutions to minimize this limitation. Amongst others, a group of respondents state that the relatively weak position of effects that are difficult to monetize is not an insolvable limitation and CBA analysts should monetize project effects as much as possible. Another group of key actors believes that one should express effects that are difficult to monetize with indicators other than money because these effects are incommensurable with travel time savings, for instance. We do not have a final position in this debate and—although we are aware that this is a very pragmatic position—we suggest sustaining the extent to which effects that are difficult to monetize are monetized on two criteria. Firstly, the share of the effect that can be monetized. Although we do not propose a sharp threshold value, if, for instance, only 10 % of the effects on biodiversity can be monetized, we would advise against the monetization of the effect because only a part of the effects on biodiversity are monetized and taken into account in the CBA report, whilst the reader might think that the total effect is included. Next, we think that the viewpoints of the relevant decision makers in a case should be decisive for the decision to monetize or not. This is in line with the argument of Nyborg (2012) who states that the more a CBA contributes to the decision makers understanding of project consequences, the more successful it is. According to Nyborg monetary valuation may help in achieving this goal when reported numbers are understandable and clarifying, but hardly helps if monetary values are perceived as provocative and confusing. We recommend future research into the extent to which this pragmatic solution has a positive influence on the use of CBA in decision-making processes. Moreover, we recommend research to evaluate the merits of expressing effects that are difficult to monetize in indicators other than in monetary terms (such as the nature value indicator, Sijtsma et al. 2011). When monetizing is too problematic, these types of rigorous and informative indicators might be good substitutes. Currently, in the Netherlands an indicator for measuring landscape effects has been developed: the Hotspot Monitor (Sijtsma et al. 2013b; De Vries et al. 2013). We think that these methods are transferable to other countries and can be promising solutions to settle the debate between fierce proponents and opponents of monetization of biodiversity and landscape. Overall, we conclude that Dutch key actors suggest two types of strategies to manage insolvable CBA limitations. The first type aspires to minimize insolvable CBA limitations, whereas the second type entirely focuses on the management of the insolvable CBA limitations. The leitmotif that can be distilled from the first type of proposed solutions is that they predominantly focus on providing more comprehensive information regarding project consequences compared to current Dutch CBA studies. This, in order to strengthen 123 Author's personal copy Transportation the usefulness of CBA in the decision-making process, for instance, by providing high quality information regarding intangible effects and distributional effects. Key actors agree that a CBA report should provide more comprehensive information regarding effects that are difficult to monetize but disagree about whether to monetize these effects with sophisticated valuation methods or providing alternative indicators (such as the nature value indicator) is the best route to take for finding a solution. As a result, the comprehensive CBA report provides high quality information regarding non-quantified effects, effects that are difficult to monetize and distributional effects alongside the monetized effects summarized in a final indicator. The second type of solutions’ leitmotif is to improve the communication of the existence of CBA limitations to non-expert users—for instance, decision makers who are important users of CBA reports—to enhance their understanding of the limitations. The proposed solutions urge CBA practitioners to be aware of the knowledge gap between them and non-expert users of CBA reports. The CBA practitioner should inform the non-expert user adequately in the CBA report about the incompleteness of the analysis, the uncertainties of effect estimations and the relatively weak position of effects that are difficult to estimate. An adequate communication of limitations means that CBA practitioners must be cautious about communicating CBA limitations in a too prominent way. Politicians will probably not consider CBA reports that endlessly emphasize the limitations of a study as useful information and certainly not as a solid basis for decision making. According to some respondents, this might eventually lead to a situation in which the use of CBA in the decision-making process will be abandoned. It is out of the scope of this paper to scrutinize what communicating CBA limitations adequately exactly means. We recommend studying this topic in further research. Based on this study we can conclude that the difficult task for CBA practitioners is to bridge the knowledge gap between them and the users of a CBA report in a ‘subtle’ way. Acknowledgments The authors wish to thank the 86 respondents for their open-heartedness. Moreover, we wish to thank four anonymous reviewers for their useful comments. Appendix 1: Respondents and affiliation at the time of the interview 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. Andre Belonje: Ministry of Infrastructure and the Environment Kirsten van den Berg: Utrecht region Peter van den Berg: Director, General Ministry of Finance Luca Bertolini: Professor, University of Amsterdam Luc Berris: Society for Preservation of Nature Monuments in the Netherlands (Natuurmonumenten) Peter Blok: Private Consultant, Rebelgroup Martijn Blom: Private Consultant, CE Delft Lauri de Boer: Private Consultant, LPBL Will Clerx: Municipality of Rotterdam Jasper Dalhuisen: Ministry of Economic Affairs, Agriculture and Innovation Karen van Dantzig: Ministry of Infrastructure and the Environment Marc Davidson: Private Consultant, CE Delft Adri Duivesteijn: Alderman, Municipality Almere Carel Eijgenraam: Researcher, Netherlands Bureau of Economic Policy Analysis (CPB) Klaas van Egmond: Professor, Utrecht University 123 Author's personal copy Transportation 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 53. 54. 55. 56. 57. 58. 59. 60. 61. 62. Terri van Dijk: Researcher, University of Groningen Henri Dijkman: Ministry of Finance Rosemarie van den Eissen: Province of Gelderland Paul Elhorst: Researcher, University of Groningen Donne Engelen: The Netherlands Society for Nature and Environment Koen Frenken: Professor, Eindhoven University of Technology Karst Geurs: Researcher, Twente University Jaron Haas: Rijkswaterstaat, Ministry of Infrastructure and the Environment Niek van der Heiden: Ministry of Infrastructure and the Environment Bart van der Heijden: Municipality of Amsterdam Wim Heijman: Professor, Wageningen University Hans Hilbers: Researcher, the Netherlands Environmental Assessment Agency (PBL) Niels Hoefsloot: Private Consultant, Decisio Arjen ‘t Hoen: Netherlands Institute for Transport Policy Analysis (KiM) Edwin Huijsman: Rijkswaterstaat, Ministry of Infrastructure and the Environment Toon van der Hoorn: Rijkswaterstaat, Ministry of Infrastructure and the Environment Bert Hof: Ministry of Finance Bas van Holst: Private Consultant, Drs. B Walter Hulsker: Private Consultant, Ecorys Ekko van Ierland: Professor, Wageningen University Martin de Jong: Researcher, Delft University of Technology Jarl Kind: Researcher, Deltares Ursula Kirchholtes: Private Consultant, Witteveen & Bos Henk Klaassen: Researcher, Erasmus University Rotterdam Jeroen Klooster: Private Consultant, Arcadis Maarten Koningsveld: Automobile Association (ANWB) Carl Koopmans: Professor, University Amsterdam Fokko Kuik: Municipality of Amsterdam Wim Korver: Private Consultant, Goudappel/Coffeng Sonja Kruitwagen: Researcher, the Netherlands Environmental Assessment Agency (PBL) Robert van Leusden: Utrecht region Ronald van der Meijs: Ministry of Infrastructure and the Environment Coen Mekers: Province of Gelderland August Mesker: Confederation of Netherlands Industry and Employers (VNO-NCW) Henk Meurs: Professor, Radboud University Nijmegen/Private Consultant, Muconsult Henk van Mourik: Ministry of Infrastructure and the Environment Roland Nijssen: Researcher, Prorail Michiel de Nooij: Private Consultant, SEO Jan Oosterhaven: Professor, University of Groningen Jan Peelen: Civil Servant, Ministry of Infrastructure and the Environment Eric Pijnappels: Private Consultant, Goudappel/Coffeng Paul Poppink: Transport and Logistics Association (TLN) Bertus Postma: Municipality of Rotterdam Aniel Ramawadh: Private Consultant, BCI Global Emiel Reiding: Ministry of Infrastructure and the Environment Gusta Renes: Researcher, the Netherlands Environmental Assessment Agency (PBL) Piet Rietveld: Professor, University Amsterdam 123 Author's personal copy Transportation 63. 64. 65. 66. 67. 68. 69. 70. 71. 72. 73. 74. 75. 76. 77. 78. 79. 80. 81. 82. 83. 84. 85. 86. Sytze Rienstra: Private Consultant, Syconomy Gerbert Romijn: Researcher, Netherlands Bureau of Economic Policy Analysis (CPB) Freddie Rosenberg: Private Consultant, RIGO Elisabeth Ruijgrok: Private Consultant, Witteveen en Bos Jan Sakko: Municipality of Rotterdam Olaf Seinen: Private Consultant, Goudappel/Coffeng Herman Stolwijk: Researcher, Netherlands Bureau of Economic Policy Analysis (CPB) Wim Spit: Private Consultant, Ecorys Edward Stigter: Ministry of Infrastructure and the Environment Lori Tavasszy: Professor, Delft University of Technology Bart Teulings: Municipality of Almere Pieter Tordoir: Professor, University of Amsterdam Bas Turpijn: Rijkswaterstaat, Ministry of Infrastructure and the Environment Hans ten Velden: Ministry of Infrastructure and the Environment Erik Verhoef: Professor, University Amsterdam Erik Verroen: Private Consultant, Twynstra Gudde Nol Verster: retired, former Private Consultant, Ecorys Johan Visser: Netherlands Institute for Transport Policy Analysis (KiM) Warren Walker: Professor, Delft University of Technology Pim Warffemius: Netherlands Institute for Transport Policy Analysis (KiM) Bart Witmond: Private Consultant, Ecorys Dik Wolfson: retired, former member of the Upper House Pauline Wortelboer: Netherlands Institute for Transport Policy Analysis (KiM) Peter Zwaneveld: Researcher, Netherlands Bureau of Economic Policy Analysis (CPB) Appendix 2: Number of respondents that mentioned a solution See Table 2. Table 2 Number of respondents that mentioned a solution Proposed solution to manage incorrect use of CBA Frequency of respondents that mentioned a solution 1.Organize effect survey meetings 4 2.Provide high quality qualitative information regarding intangible effects 9 3.Carry out ex-post analyses 9 4.Monetize as much as possible 17 5.Use indicators other than money to express effects that are difficult to monetize 15 6. Monetize effects in CBA studies when possible. However, when monetizing leads to severe criticism and frustrations, one should not monetize 6 7. Referent assesses the position of effects that are difficult to monetize in the CBA report 1 8. Explicitly discuss the way monetized and non-monetized effects relate to each other in the summary of the CBA report 1 123 Author's personal copy Transportation Table 2 continued Proposed solution to manage incorrect use of CBA Frequency of respondents that mentioned a solution 9. Implement a ‘showstopper’ procedure in the CBA analysis for effects that are difficult to monetize and that are irreversible 1 10. CBA reports should be supplemented by an analysis of distributional effects 8 11.Enhance communication of incompleteness in CBA reports 9 12.Enhance communication of uncertainty in CBA reports 39 13. Estimate the project effects for at least one flexible investment strategy in a CBA study 8 14. Communicate that readers should also take effects that are difficult to monetize into account 5 References Ackerman, F., Heinzerling, L.: Priceless: on knowing the price of everything and the value of nothing. New Press, New York (2004) Ambrasaite, I., Barfod, M.B., Salling, K.B.: MCDA and risk analysis in transport infrastructure appraisal: the rail Baltica case. Proc. Soc. Behav. Sci. 20, 944–953 (2011) Atkinson, G., Mourato, S.: Environmental cost-benefit analysis. Annu. Rev. Environ. Resour. 33, 317–344 (2008) Barfod, M.B., Salling, K.B., Leleur, S.: Composite Decision Support by Combining Cost-Benefit and MultiCriteria Decision Analysis. Decis. Support Syst. 51, 167–175 (2011) Beattie, R.B.: Everything you already know about EIA (but don’t often admit). Environ. Impact Assess. Rev. 15(2), 109–114 (1995) Börjesson, M., Eliasson, J., Lundberg, M.:Is CBA ranking for transport investment robust? J. Transp. Econ. Policy Buck Consult. Int. 2002. Evaluatie OEEI-leidraad (In Dutch). Buck Consultants, The Hague (2013) [Epub ahead of print] Boadway, R.: Principles of cost-benefit analysis. Public Policy Rev. 2(1), 1–42 (2006) Campbell, H.F., Brown, R.P.C.: Benefit-Cost Analysis: Financial and Economic Appraisal Using Spreadsheets. Cambridge University Press, Cambridge (2003) Eijgenraam, C.J.J., Koopmans, C.C., Tang, P.J.G., Verster, A.C.P.: Evaluation of Infrastructural Projects; Guide for Cost-Benefit Analysis, Sections I and II, CPB. The Hague, NEI (2000). (Changed Name to ECORYS), Rotterdam Eliasson, J., Lundberg, M.: Do cost-benefit analyses influence transport investment decisions? Experiences from the Swedish transport investment plan 2010–2021. Transp. Rev. 32(1), 29–48 (2012) Flyvbjerg, B., Bruzelius, N., Rothengatter, W.: Megaprojects and Risk—An Anatomy of Ambition. Cambridge University Press, Cambridge (2003) Flyvbjerg, B., Skamris Holm, M.K., Buhl, S.L.: How (in) accurate are demand forecasts in public works projects? The case of transportation. J. Am. Plann. Assoc. 71, 131–146 (2005) Grant-Muller, S., Mackie, P., Nellthorp, J., Pearman, A.: Economic appraisal of European transport projects—The state of the art revisited. Transp. Rev. 21(2), 237–261 (2001) Hagen, K. P.: Official Norwegian Reports NOU 2012: 16. Review from a committee appointed by Royal Decree of 18 February 2011, Oslo (2012) Hayashi, Y., Morisugi, H.: International comparison of background concept and methodology of transportation project appraisal. Transp. Policy 7(1), 73–88 (2000) Jong, D.M., Geerlings, H.: Exposing weaknesses in interactive planning: the remarkable return of comprehensive policy analysis in The Netherlands. Impact Assess. Proj. Apprais. 21(4), 281–291 (2003) Jong, G.C., Daly, A.J., Pieters, M., Miller, S., Plasmeijer, R., Hofman, F.: Uncertainty in traffic forecasts: literature review and new results for The Netherlands. Transportation 34, 375–395 (2007) 123 Author's personal copy Transportation Kelman, S.: Cost-benefit analysis: an ethical critique. In: Donaldson, T., Werhane, P.H., Cording, M. (eds.) Ethical Issues in Business: A Philosophical Approach, pp. 559–564. Prentice Hall, Upper Saddle River (2002) Mackie, P.: Cost-Benefit Analysis in Transport: A UK Perspective. International Transport Forum, Mexico (2010) Mackie, P., Preston, J.: Twenty-one sources of error and bias in transport project appraisal. Transp. Policy 5(1), 1–7 (1998) Mackie, P., Worsley, T.: International Comparisons of Transport Appraisal Practice, Overview Report. Institute for Transport Studies, University of Leeds, Leeds (2013) Martens, K.: Substance precedes methodology: on cost-benefit analysis and equity. Transportation 38, 959–974 (2011) Mishan, E.J.: Elements of Cost-Benefit Analysis. George Allen and Unwin, London (1976) Mishan, E.J.: Cost-Benefit Analysis. Allen and Unwin, London (1988) Mouter, N., Annema, J.A., Van Wee, B.: Maatschappelijke Kosten- en Batenanalyse inhoudelijk geëvalueerd. NICIS, The Hague (2012) Mouter, N., Annema, J.A., Van Wee, B.: Ranking the substantive problems in the Dutch Cost-Benefit Analysis practice. Transp. Res. Part A 49, 241–255 (2013a) Mouter, N., Annema, J.A., van Wee, B.: Attitudes towards the role of cost-benefit analysis in the decisionmaking process for spatial-infrastructure projects: a Dutch case study. Transp. Res. Part A 58, 1-18 (2013b) Mouter, N., Holleman, M., Calvert, S., Annema, J.A.: A personality and social psychology perspective on communicating uncertainty in cost-benefit analysis. In: Proceedings of International Conference on Uncertainty in Transport Evaluation. 16–18 Sept 2013, Copenhagen (2013c) Naess, P.: Cost-benefit analysis of transportation investments. Neither critical nor realistic. J. Crit. Realis. 5(1), 32–60 (2006) Naess, P., Strand, A.: What kinds of traffic forecasts are possible? J. Crit. Realis. 11(3), 277–295 (2012) Nicolaisen, M. S.: Forecasts: Fact or Fiction? Uncertainty and Inaccuracy in Transport Project Evaluation. Department of Develpment and Planning, Aalborg University (2012) Nyborg, K.: The Ethics and Politics of Environmental Cost-Benefit Analysis. Routledge, London (2012) Odgaard, T., Kelly, C., Laird, J.: Current practice in project appraisal in Europe, In: Proceedings of the European Transport Conference. 3–5 Oct 2005, Strasbourg: Association for European Transport (2005) Rietveld, P., Rouwendal, J., Vlist, A.J.: Equity issues in the evaluation of transport policies and transport infrastructure projects. In: Geenhuizen, M., van Reggiani, A., Rietveld, P. (eds.) Policy Analysis of Transport Networks, pp. 19–36. Ashgate, Aldershot (2007) Reuer, J. J., Tong, T. W.: Real options in strategic management. In: Real options Theory Adv. Strateg. Manag. 22, 3-28. (2007) Salling, K.B., Banister, D.: Assessment of large transport infrastructure project: the CBA-DK model. Transp. Res. Part A 43(9), 800–813 (2009) Self, P.: Nonsense on stilts: Cost-Benefit Analysis and the Roskill Commission. Polit.Quart. 41(3), 249–260 (1970) Sen, A.K.: The discipline of cost-benefit analysis. J.Legal Stud. 29(2), 931–952 (2000) Sijtsma, F.J., Van der Heide, C.M., van Hinsberg, A.: Biodiversity and decision-support: integrating CBA and MCA. In: Hull, A., Alexander, E., Khakee, A., Woltjer, J. (eds.) Evaluation for Participation and Sustainability in Planning, vol. 9, pp. 197–218. Routledge, London (2011) Sijtsma, F.J., Van der Heide, C.M., Van Hinsberg, A.: Beyond monetary measurement: how to evaluate projects and policies using the ecosystem services framework. Environ. Sci. Policy 32, 14–25 (2013a) Sijtsma, F.J., Farjon, H.S., van Tol, A., van Hinsberg, A., van Kampen, P., Buijs, A.: Evaluation of landscape changes—enriching the economist’s toolbox with the hotspotindex. In: Heijman, W., Van der Heide, C.M.J. (eds.) The Economic Value of Landscapes, vol. 8, pp. 136–164. Routledge, London (2013b) Sijtsma, F.J.: Project evaluation, sustainability and accountability—combining cost-benefit analysis (CBA) and multi-criteria analysis (MCA). PhD Thesis, University of Groningen. Stichting REG, no. 27. Groningen (2006) Thomopoulos, N., Grant-Muller, S., Tight, M.R.: Incorporating equity considerations in transport infrastructure evaluation: current practice and a proposed methodology. Eval. Progr. Plan. 32(4), 351–359 (2009) Thomopoulos, N., Grant-Muller, S.: Incorporating equity as part of the wider impacts in transport infrastructure assessment: an application of the SUMINI approach. Transportation 40(2), 315–345 (2013) 123 Author's personal copy Transportation Thompson, M.M., Naccarato, M.E., Parker, K.C.H., Moskowitz, G.B.: The personal need for structure (PNS) and personal fear of invalidity (PFI) measures: historical perspectives, present applications and future directions. In: Moskowitz, G.B. (ed.) Cognitive Social Psychology: The Princeton Symposium on the Legacy and Future of Social Cognition, pp. 19–39. Lawrence Erlbaum Associates Inc., Mahwah, NJ (2001) Vries. S, de., Buijs, A., Langers, F., Farjon, H.S., Hinsberg, A, van., Sijtsma, F. J.:Measuring the attractiveness of Dutch landscapes: identifying national hotspots using Google Maps. Applied Geography (2013) [Epub ahead of print] Weber, R.P.: Basic Content Analysis, 2nd edn. Newbury Park, California (1990) Wee, V.B.: How suitable is CBA for the ex-ante evaluation of transport projects and policies? A discussion from the perspective of ethics. Transp. Policy 19(1), 1–7 (2012) Welsh, M., Williams, H.: The sensitivity of transport investments benefits to the evaluation of small traveltime savings. J. Transp. Econ. Policy 31, 231–254 (1997) Niek Mouter is a postdoctoral researcher at TU Delft’s Transport and Logistics Group. His research primarily focuses on the incorporation of ethical notions and theories in cost-benefit analysis. He has published papers in Transportation Research Part A and Transport Policy. Jan Anne Annema is assistant professor Transport Policy at TU Delft’s Transport and Logistics Group. His education and research focuses on transport policy in general with focus on policy evaluation methods and policy implementation issues. Bert van Wee is professor of Transport Policy at Delft University of Technology’s Transport and Logistics Group, and scientific director of TRAIL Research School. His research focuses on accessibility, (evaluation of) large infrastructure projects, the environment, safety, and ethics. 123