Abstract
Numerous multicriteria decision aiding (MCDA) methods have been developed over the last decades and are now applied in various domains, sometimes using facilitated group workshops to create models. These models are all designed to improve decision processes. However, the lack of follow-up and post-project evaluations limit the understanding of how the participants experienced the group workshops and how the results were subsequently used within the organization. This is in contrast with the public participation research field, where a rich literature was developed for a posteriori evaluation of projects. Based on this literature, our research proposes a framework to evaluate, ex-post, MCDA projects. In order to illustrate this framework, we apply it to an MCDA project in Quebec City where a spatial decision support system to prioritize the redesign of streets as Complete Streets was built. Individual interviews were conducted with the Quebec City professionals that currently use, were leaders of the project, or have participated in the development of the decision support system. This research has identified that the need for change of practices within the workplace, communication problems, and the requirement for multidisciplinary work were at the root of the various challenges encountered during the workshops. Based on our experience, we propose some lessons learned and potential solutions that can enhance the body of literature in MCDA.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and material
To respect the consent form that all participants signed, the individual results from the participant interviews such as the transcripts are not available, and participants’ quotes were anonymized in this paper.
Notes
The term artifacts (Simon 1996) will be used in this paper as a global term that refers to the set of new objects (models, tools, or decision support systems) that are created and designed in an MCDA or OR project (Dresch et al. 2014). In more particular cases, the specific term such as “the model” or “the tool” will be used.
References
Ackermann F, Yearworth M, White L (2018) Micro-processes in Group Decision and Negotiation: practices and routines for supporting decision making. Group Decis Negot 27:709–713. https://doi.org/10.1007/s10726-018-9590-x
Ackoff RL (1962) Some unsolved problems in problem solving. Oper Res Q 13:12. https://doi.org/10.2307/3007575
Ackoff RL (1979a) The future of operational research is past. J Oper Res Soc 30:93. https://doi.org/10.2307/3009290
Ackoff RL (1979b) Resurrecting the future of operational research. J Oper Res Soc 30:189. https://doi.org/10.2307/3009600
Andersen DF, Richardson GP (1997) Scripts for group model building. Syst Dyn Rev 13:107–129. https://doi.org/10.1002/(SICI)1099-1727(199722)13:2%3c107::AID-SDR120%3e3.0.CO;2-7
Bana e Costa CA, De Corte J-M, Vansnick J-C (2005) M-MACBETH. Version 2.4.0. BANA Consulting
Bana e Costa CA, De Corte J-M, Vansnick J-C (2016) On the Mathematical Foundations of MACBETH. In: Greco S, Ehrgott M, Figueira JR (eds) Multiple Criteria Decision Analysis. Springer, New York, NY, pp 421–463
Banville C, Landry M, Martel J-M, Boulaire C (1998) A stakeholder approach to MCDA. Syst Res Behav Sci 15:15–32. https://doi.org/10.1002/(SICI)1099-1743(199801/02)15:1%3c15::AID-SRES179%3e3.0.CO;2-B
Barcus A, Montibeller G (2008) Supporting the allocation of software development work in distributed teams with multi-criteria decision analysis. Omega 36:464–475. https://doi.org/10.1016/j.omega.2006.04.013
Becker KH (2016) An outlook on behavioural OR—three tasks, three pitfalls, one definition. Eur J Oper Res 249:806–815. https://doi.org/10.1016/j.ejor.2015.09.055
Behzadian M, Kazemzadeh RB, Albadvi A, Aghdasi M (2010) PROMETHEE: a comprehensive literature review on methodologies and applications. Eur J Oper Res 200:198–215. https://doi.org/10.1016/j.ejor.2009.01.021
Bouyssou D, Perny P, Pirlot M et al (1993) A manifesto for the new MCDA era. J Multi-Criteria Decis Anal 2:125–127. https://doi.org/10.1002/mcda.4020020302
Braune I, Pinkwart A, Reeg M (2009) Application of multi-criteria analysis for the evaluation of sustainable energy systems - A review of recent literature. In: Proceedings of the 5th Conference on Sustainable Development of Energy, Water and Environment Systems. Dubrovnik, Croatia, p 13
Brown G, Chin SYW (2013) Assessing the effectiveness of public participation in neighbourhood planning. Plan Pract Res 28:563–588. https://doi.org/10.1080/02697459.2013.820037
Browne D, Ryan L (2011) Comparative analysis of evaluation techniques for transport policies. Environ Impact Assess Rev 31:226–233. https://doi.org/10.1016/j.eiar.2010.11.001
Chakhar S, Mousseau V (2017) Multicriteria spatial decision support systems. In: Shekhar S, Xiong H, Zhou X (eds) Encyclopedia of GIS. Springer International Publishing, Cham, pp 1404–1411
Chess C (2000) Evaluating Environmental Public Participation: Methodological Questions. J Environ Plan Manag 43:769–784. https://doi.org/10.1080/09640560020001674
Choi BCK, Pak AWP (2005) A catalog of biases in questionnaires. Prev Chronic Dis 2:13
Churchman CW (1970) Operations research as a profession. Manag Sci 17:B-37-B-53. https://doi.org/10.1287/mnsc.17.2.B37
Communauté métropolitaine de Québec (2013) Plan métropolitain d’aménagement et de développement du territoire de la Communauté métropolitaine de Québec
CORS (2019) Practice Prize Competition. In: CORS. https://www.cors.ca/?q=content/practice-prize-competition. Accessed 26 Feb 2020
Damart S (2010) A cognitive mapping approach to organizing the participation of multiple actors in a problem structuring process. Group Decis Negot 19:505–526. https://doi.org/10.1007/s10726-008-9141-y
de Gooyert V, Rouwette E, van Kranenburg H, Freeman E (2017) Reviewing the role of stakeholders in operational research: a stakeholder theory perspective. Eur J Oper Res 262:402–410. https://doi.org/10.1016/j.ejor.2017.03.079
Decision Analysis Society (2020) DAS Practice Award. In: INFORMS. https://www.informs.org/Recognizing-Excellence/Community-Prizes/Decision-Analysis-Society/DAS-Practice-Award. Accessed 26 Feb 2020
D’Este G (2009) Capturing different viewpoints in multi-criteria analysis. In: Proceedings of the 32th Australasian Transport Research Forum. p 11p (session Wed 3A)
Dresch A, Lacerda DP, Antunes Jr. JAV (2014) Design science—the science of the artificial. In: Design science research. Springer, Cham, New York
Eden C (1995) On evaluating the performance of ‘wide-band’ GDSS’s. Eur J Oper Res 81:302–311. https://doi.org/10.1016/0377-2217(93)E0241-O
Eden C, Ackermann F (1996) “Horses for courses”: a stakeholder approach to the evaluation of GDSSs. Group Decis Negot 5:501–519
ESRI (2014) ArcMap. Version 10.2.2. ESRI
Ferreira F, Santos S, Rodrigues P (2011) Adding value to bank branch performance evaluation using cognitive maps and MCDA: a case study. J Oper Res Soc 62:1320–1333
Franco LA, Greiffenhagen C (2018) Making OR practice visible: Using ethnomethodology to analyse facilitated modelling workshops. Eur J Oper Res 265:673–684. https://doi.org/10.1016/j.ejor.2017.08.016
Franco LA, Hämäläinen RP (2016) Behavioural operational research: returning to the roots of the OR profession. Eur J Oper Res 249:791–795. https://doi.org/10.1016/j.ejor.2015.10.034
Franco LA, Montibeller G (2010) Facilitated modelling in operational research. Eur J Oper Res 205:489–500. https://doi.org/10.1016/j.ejor.2009.09.030
Franco LA, Nielsen MF (2018) Examining group facilitation in situ: the use of formulations in facilitation practice. Group Decis Negot. https://doi.org/10.1007/s10726-018-9577-7
Fung A (2006) Varieties of participation in complex governance. Public Adm Rev 66:66–75. https://doi.org/10.1111/j.1540-6210.2006.00667.x
Geldermann J, Bertsch V, Treitz M et al (2009) Multi-criteria decision support and evaluation of strategies for nuclear remediation management. Omega 37:238–251. https://doi.org/10.1016/j.omega.2006.11.006
Genard J-L, Pirlot M (2002) Multi-criteria decision-aid in a philosophical perspective. In: Bouyssou D, Jacquet-Lagrèze E, Perny P et al (eds) Aiding decisions with multiple criteria: essays in honor of Bernard Roy. Springer, US, pp 89–117
Govindan K, Jepsen MB (2016) ELECTRE: acomprehensive literature review on methodologies and applications. Eur J Oper Res 250:1–29. https://doi.org/10.1016/j.ejor.2015.07.019
Greene R, Luther JE, Devillers R, Eddy B (2010) An approach to GIS-based multiple criteria decision analysis that integrates exploration and evaluation phases: Case study in a forest-dominated landscape. For Ecol Manag 260:2102–2114. https://doi.org/10.1016/j.foreco.2010.08.052
Henao F, Franco LA (2016) Unpacking multimethodology: impacts of a community development intervention. Eur J Oper Res 253:681–696. https://doi.org/10.1016/j.ejor.2016.02.044
Hess P (2009) Avenues or arterials: the struggle to change street building practices in Toronto, Canada. J Urban Des 14:1–28. https://doi.org/10.1080/13574800802451049
Hovmand PS, Andersen DF, Rouwette E et al (2012) Group model-building ‘Scripts’ as a collaborative planning tool: scripts as a collaborative planning tool. Syst Res Behav Sci 29:179–193. https://doi.org/10.1002/sres.2105
Jones NA, Perez P, Measham TG et al (2009) Evaluating participatory modeling: developing a framework for cross-case analysis. Environ Manage 44:1180–1195. https://doi.org/10.1007/s00267-009-9391-8
Keeney RL (1996) Value-focused thinking: identifying decision opportunities and creating alternatives. Eur J Oper Res 92:537–549. https://doi.org/10.1016/0377-2217(96)00004-5
Keren G, de Bruin WB (2005) On the Assessment of Decision Quality: Considerations Regarding Utility, Conflict and Accountability. In: Hardman D, Macchi L (eds) Thinking: Psychological Perspectives on Reasoning, Judgment and Decision Making. John Wiley & Sons, Ltd, Chichester, UK, pp 347–363
Kunc M, Malpass J, White L (eds) (2016) Behavioral operational research: theory, methodology and practice. Palgrave Macmillan, London
Landry M, Banville C, Oral M (1996) Model legitimisation in operational research. Eur J Oper Res 92:443–457. https://doi.org/10.1016/0377-2217(96)00003-3
Leleur S (2017) Green Decision Making: How systemic Planning can support Strategic Decision Making for Sustainable Transport Development. Technical University of Denmark, Kongens Lingsby, Denmark
Lienert J, Koller M, Konrad J et al (2011) Multiple-criteria decision analysis reveals high stakeholder preference to remove pharmaceuticals from hospital wastewater. Environ Sci Technol 45:3848–3857. https://doi.org/10.1021/es1031294
Malczewski J (2006) GIS-based multicriteria decision analysis: a survey of the literature. Int J Geogr Inf Sci 20:703–726. https://doi.org/10.1080/13658810600661508
Marleau Donais F, Abi-Zeid I, Lavoie R (2017a) Building a shared model for multi-criteria group decision making. In: Schoop M, Kilgour M (eds) Group decision and negotiation. A Socio-Technical Perspective. Springer, Cham, pp 175–186
Marleau Donais F, Abi-Zeid I, Lavoie R (2017b) A loose-coupling integration of the MACBETH approach in ArcGIS. In: Linden I, Mareschal B, Liu S et al (eds) Proceedings of the 2017 international conference on decision support system technology. EWG-DSS, Namur, Belgique, pp 125–131
Marleau Donais F, Abi-Zeid I, Waygood EOD, Lavoie R (2019) Assessing and ranking the potential of a street to be redesigned as a complete street: a multi-criteria decision aiding approach. Transp Res Part Policy Pract 124:1–19. https://doi.org/10.1016/j.tra.2019.02.006
Marttunen M, Hämäläinen RP (2008) The decision analysis interview approach in the collaborative management of a large regulated water course. Environ Manage 42:1026–1042. https://doi.org/10.1007/s00267-008-9200-9
Marttunen M, Lienert J, Belton V (2017) Structuring problems for Multi-Criteria Decision Analysis in practice: a literature review of method combinations. Eur J Oper Res. https://doi.org/10.1016/j.ejor.2017.04.041
Matheson JE (2013) Decision analysis in practice. In: Gass SI, Fu MC (eds) Encyclopedia of operations research and management science, 3rd edn. Springer, Boston
McCann B (2013) Completing our streets: the transition to safe and inclusive transportation networks. Island Press, Washington
McCardle-Keurentjes M, Rouwette EAJA (2018) Asking questions: a Sine Qua non of facilitation in decision support? Group Decis Negot 27:757–788. https://doi.org/10.1007/s10726-018-9573-y
McCartt AT, Rohrbaugh J (1995) Managerial openness to change and the introduction of GDSS: explaining initial success and failure in decision conferencing. Organ Sci 6:569–584. https://doi.org/10.1287/orsc.6.5.569
Midgley G, Cavana RY, Brocklesby J et al (2013) Towards a new framework for evaluating systemic problem structuring methods. Eur J Oper Res 229:143–154. https://doi.org/10.1016/j.ejor.2013.01.047
Mingers J, Brocklesby J (1997) Multimethodology: Towards a framework for mixing methodologies. Omega 25:489–509. https://doi.org/10.1016/S0305-0483(97)00018-2
Montibeller G (2005) From (and To) a New Generation of Multi-Criteria Decision Analysts: An Introduction to the Field and a Personal View on its Future. In: Keynote Papers of YOR 14 Conference. Birmingham, p 14
Munda G (2008) Social multi-criteria evaluation for a sustainable economy. Springer, New York
Mustajoki J, Hämäläinen RP, Marttunen M (2004) Participatory multicriteria decision analysis with Web-HIPRE: a case of lake regulation policy. Environ Model Softw 19:537–547. https://doi.org/10.1016/j.envsoft.2003.07.002
Paillé P, Mucchielli A (2016a) L’analyse qualitative en sciences humaines et sociales, 4e edn. Armand Colin, Paris, France
Paillé P, Mucchielli A (2016b) L’analyse thématique. In: L’analyse qualitative en sciences humaines et sociales, 4e edn. Armand Colin, Paris, France, pp 235–317
Papamichail KN, Alves G, French S et al (2007) Facilitation practices in decision workshops. J Oper Res Soc 58:614–632. https://doi.org/10.1057/palgrave.jors.2602373
Paschetta E, Tsoukiàs A (2000) A real-world MCDA application: evaluating software. J Multi-Criteria Decis Anal 9:205–226. https://doi.org/10.1002/1099-1360(200009)9:5%3c205::AID-MCDA277%3e3.0.CO;2-O
Phillips LD (2007) Decision conferencing. In: von Winterfeldt D (ed) Edwards W, Miles RFJr. Advances in Decision Analysis. Cambridge University Press, Cambridge, pp 375–399
Phillips LD (2011) Group dynamics processes for improved decision making. In: Wiley Encyclopedia of Operations Research and Management Science. John Wiley & Sons, Inc., Hoboken, NJ, USA
Phillips LD, Phillips MC (1993) Facilitated work groups: theory and practice. J Oper Res Soc 44:533–549. https://doi.org/10.1038/sj/jors/0440601
QSR International (2019) N-Vivo. Version 12. QSR International. URL https://www.qsrinternational.com/nvivo/nvivo-products/nvivo-12-plus
Quinet E, Meunier D (2012) CBA § Darwin: the case of transport infrastructure in France. Proc Soc Behav Sci 48:3051–3064. https://doi.org/10.1016/j.sbspro.2012.06.1272
Rouwette EAJA, Vennix JAM, Felling AJA (2009) On evaluating the performance of problem structuring methods: an attempt at formulating a conceptual model. Group Decis Negot 18:567–587. https://doi.org/10.1007/s10726-007-9100-z
Rouwette EAJA, Vennix JAM, van Mullekom T (2002) Group model building effectiveness: a review of assessment studies. Syst Dyn Rev 18:5–45. https://doi.org/10.1002/sdr.229
Rowe G, Frewer LJ (2000) Public participation methods: a framework for evaluation. Sci Technol Hum Values 25:3–29. https://doi.org/10.1177/016224390002500101
Rowe G, Frewer LJ (2004) Evaluating public-participation exercises: a research agenda Sci Technol Hum Values 29:512–556. https://doi.org/10.1177/0162243903259197
Salo A, Hämäläinen RP (2010) Multicriteria decision analysis in group decision processes. In: Handbook of group decision and negotiation. Springer, pp 269–283
Schuwirth N, Reichert P, Lienert J (2012) Methodological aspects of multi-criteria decision analysis for policy support: a case study on pharmaceutical removal from hospital wastewater. Eur J Oper Res 220:472–483. https://doi.org/10.1016/j.ejor.2012.01.055
Scott RJ, Cavana RY, Cameron D (2013) Evaluating immediate and long-term impacts of qualitative group model building workshops on participants’ mental models: Group Model Building and Participants’ Mental Models. Syst Dyn Rev 29:216–236. https://doi.org/10.1002/sdr.1505
Simon HA (1996) The sciences of the artificial, 3rd edn. MIT Press, Cambridge
Smart Growth America, Coalition NCS (2018) The Best Complete Streets Initiatives of 2017. Smart Growth America, Washington, D.C.
Statistique Canada (2017) Québec, V [Subdivision de recensement], Québec et Québec, TÉ [Division de recensement], Québec (tableau). Profil du recensement. http://www12.statcan.gc.ca/census-recensement/2016/dp-pd/prof/details/page.cfm?Lang=F&Geo1=CSD&Code1=2423027&Geo2=CD&Code2=2423&Data=Count&SearchText=quebec&SearchType=Begins&SearchPR=01&B1=All&TABID=1. Accessed 17 May 2017
Tavella E, Papadopoulos T (2015a) Expert and novice facilitated modelling: a case of a Viable System Model workshop in a local food network. J Oper Res Soc 66:247–264. https://doi.org/10.1057/jors.2013.187
Tavella E, Papadopoulos T (2015b) Novice facilitators and the use of scripts for managing facilitated modelling workshops. J Oper Res Soc 66:1967–1988. https://doi.org/10.1057/jors.2015.7
Venable J, Pries-Heje J, Baskerville R (2016) FEDS: a framework for evaluation in design science research. Eur J Inf Syst 25:77–89. https://doi.org/10.1057/ejis.2014.36
Ville de Québec (2017) Rues conviviales: Une approche intégrée de conception de rues pour améliorer la qualité de vie urbaine
White L (2006) Evaluating problem-structuring methods: developing an approach to show the value and effectiveness of PSMs. J Oper Res Soc 57:842–855. https://doi.org/10.1057/palgrave.jors.2602149
Yates JF, Veinott ES, Patalano AL (2003) Hard decisions, bad decisions: on decision quality and decision aiding. In: Schneider SL, Shanteau J (eds) Emerging perspectives on judgment and decision research. Cambridge University Press, Cambridge, pp 13–63
Acknowledgements
The authors wish to thank all the interviewees from Ville de Québec for their input and valuable time.
Funding
Financial support was provided for this project through scholarships by the Social Sciences and Humanities Research Council of Canada (Grant 752–2017-2758), the Fonds de Recherche du Québec—Nature et Technologies (Grant 207792), the Institute Hydro-Québec in Environment, Development and Society and the Transportation Association of Canada.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethical approval
Ethics and research committee of Laval University (no 2018-275/ 17-12-2018).
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Marleau Donais, F., Abi-Zeid, I., Waygood, E.O.D. et al. A Framework for Post-Project Evaluation of Multicriteria Decision Aiding Processes from the Stakeholders’ Perspective: Design and Application. Group Decis Negot 30, 1161–1191 (2021). https://doi.org/10.1007/s10726-021-09753-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10726-021-09753-y