Abstract
Composite indicators have been increasingly recognized as a useful tool for performance monitoring, benchmarking comparisons and public communication in a wide range of fields. The usefulness of a composite indicator depends heavily on the underlying data aggregation scheme where multiple criteria decision analysis (MCDA) is commonly used. A problem in this application is the determination of an appropriate MCDA aggregation method. Of the many criteria for comparing MCDA methods, the Shannon-Spearman measure (SSM) is one that compares alternative MCDA aggregation methods in constructing composite indicators based on the information loss concept. This paper assesses the effectiveness of the SSM using Monte Carlo approach-based uncertain analysis and variance-based sensitivity analysis techniques. It is found that most of the variation in the SSM arises from the uncertainty in choosing an aggregation method. Therefore, the SSM can be considered as an effective measure for comparing MCDA aggregation methods in constructing composite indicators. We also use the SSM to evaluate five MCDA aggregation methods in constructing composite indicators and present the findings.
Similar content being viewed by others
Notes
Although there are many different interpretations of criteria weights in MCDA (e.g., Choo et al. 1999), the most commonly used ones in constructing CIs appear to be the “trade-off ratios” and “coefficients of importance” (Munda 2005; Nardo et al. 2005). In this study the criteria weights are assumed to be exogenous and will not be discussed in detail.
There is not a universally agreed upon definition of “validity.” Different researchers/analysts may interpret it differently, such as predictive validity, estimative validity, methodological validity, construct validity and convergent validity (Hobbs 1986). The various definitions of “validity” arise from the different purposes of MCDA methods in application. In this study, the validity of SSM refers to whether most of its variation can be explained by the change in the MCDA aggregation method.
References
Booysen, F. (2002). An overview and evaluation of composite indices of development. Social Indicators Research, 59, 115–151. doi:10.1023/A:1016275505152.
Chan, K., Tarantola, S., Saltelli, A., & Sobol’, I. M. (2000). Variance based methods. In A. Saltelli, K. Chan, & E. Scott (Eds.), Sensitivity analysis (pp. 167–197). New York: Wiley.
Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2, 429–444. doi:10.1016/0377-2217(78)90138-8.
Cherchye, L., Lovell, C. A. K., Moesen, W., & van Puyenbroeck, T. (2007a). One market, one number? A composite indicator assessment of EU internal market dynamics. European Economic Review, 51, 749–779. doi:10.1016/j.euroecorev.2006.03.011.
Cherchye, L., Moesen, W., Rogge, N., & van Puyenbroeck, T. (2007b). An introduction to ‘benefit of the doubt’ composite indicators. Social Indicators Research, 82, 111–145. doi:10.1007/s11205-006-9029-7.
Cherchye, L., Moesen, W., Rogge, N., van Puyenbroeck, T., Saisana, M., Saltelli, A., et al. (2008). Creating composite indicators with DEA and robustness analysis: The case of the Technology Achievement Index. Journal of the Operational Research Society, 59, 239–251.
Choo, E. U., Schoner, B., & Wedley, W. C. (1999). Interpretation of criteria weights in multicriteria decision making. Computers & Industrial Engineering, 37, 527–541. doi:10.1016/S0360-8352(00)00019-X.
Despotis, D. K. (2005a). A reassessment of the human development index via data envelopment analysis. The Journal of the Operational Research Society, 56, 96–980. doi:10.1057/palgrave.jors.2601927.
Despotis, D. K. (2005b). Measuring human development via data envelopment analysis: The case of Asia and the Pacific. Omega, 33, 385–390. doi:10.1016/j.omega.2004.07.002.
Diakoulaki, D., Mavrotas, G., & Papayannakis, L. (1995). Determining objective weights in multiple criteria problems: The CRITIC method. Computers & Operations Research, 22, 763–770. doi:10.1016/0305-0548(94)00059-H.
Diaz-Balteiro, L., & Romero, C. (2004). In search of a natural systems sustainability index. Ecological Economics, 49, 401–405. doi:10.1016/j.ecolecon.2004.02.005.
Duckstein, L., Gershon, M., & Mcaniff, R. (1982). Model selection in multiobjective decision making for river basin planning. Advances in Water Resources, 5, 178–184. doi:10.1016/0309-1708(82)90040-9.
Ebert, U., & Welsch, H. (2004). Meaningful environmental indices: A social choice approach. Journal of Environmental Economics and Management, 47, 270–283. doi:10.1016/j.jeem.2003.09.001.
Efron, B., & Tibshirani, R. (1993). An introduction to the Bootstrap. New York: Chapman and Hall.
Esty, D. C., Levy, M. A., Srebotnjak, T., & de Sherbinin, A. (2005). 2005 environmental sustainability index: Benchmarking national environmental stewardship. New Haven, Conn: Yale Center for Environmental Law & Policy.
Figueira, J., Greco, S., & Ehrgott, M. (Eds.). (2005). Multiple criteria decision analysis: State of the art surveys. New York: Springer.
Gershon, M., & Duckstein, L. (1984). A procedure for selection of a multiobjective technique with application to water and mineral resources. Applied Mathematics and Computation, 14, 245–271. doi:10.1016/0096-3003(84)90024-9.
Guitouni, A., & Martel, J. M. (1998). Tentative guidelines to help choosing an appropriate MCDA method. European Journal of Operational Research, 109, 501–521. doi:10.1016/S0377-2217(98)00073-3.
Hobbs, B. F. (1986). What can we learn from experiments in multiobjective decision analysis? IEEE Transactions on Systems, Man and Cybernetics, 16, 384–394. doi:10.1109/TSMC.1986.4308970.
Hobbs, B. F., Chankong, V., Hamadeh, W., & Stakhiv, E. Z. (1992). Does choice of multicriteria method matter? An experiment in water resources planning. Water Resources Research, 28, 1767–1779. doi:10.1029/92WR00712.
Hoskins, B.L., & Mascherini, M. Measuring active citizenship through the development of a composite indicator. Social Indicators Research, in press. doi:10.1007/s11205-008-9271-2.
Lee, G. K. L., & Chan, E. H. W. (2008). The analytic hierarchy process (AHP) approach for assessment of urban renewal proposals. Social Indicators Research, 89, 155–168. doi:10.1007/s11205-007-9228-x.
Lun, G., Holzer, D., Tappeiner, G., & Tappeiner, U. (2006). The stability of rankings derived from composite indicators: Analysis of the “Il Sole 24 Ore” quality of life report. Social Indicators Research, 77, 307–331. doi:10.1007/s11205-005-4505-z.
Munda, G. (2005). “Measuring sustainability”: A multi-criterion framework. Environment, Development and Sustainability, 7, 117–134. doi:10.1007/s10668-003-4713-0.
Munda, G., & Nardo, M. (2003). On the methodological foundations of composite indicators used for ranking countries. Paper presented at the First OECD/JRC Workshop on Composite Indicators of Country performance, Ispra, JRC.
Murias, P., de Miguel, J. C., & Rodriguez, D. (2008). A composite indicator for university quality assessment: The case of Spanish higher education system. Social Indicators Research, 89, 129–146. doi:10.1007/s11205-007-9226-z.
Murias, P., Martinez, F., & de Miguel, C. (2006). An economic wellbeing index for the Spanish provinces: A data envelopment analysis approach. Social Indicators Research, 77, 395–417. doi:10.1007/s11205-005-2613-4.
Nardo, M., Paisana, M., Saltelli, A., Tarantola, S., Hoffman, A., & Giovannini, E. (2005). Handbook on constructing composite indicators: Methodology and user Guide. OECD Statistics Working Paper 2005/3, OECD Statistics Directorate.
Olson, D. L. (2001). Comparison of three multicriteria methods to predict known outcomes. European Journal of Operational Research, 130, 576–587. doi:10.1016/S0377-2217(99)00416-6.
Saisana, M., Saltelli, A., & Tarantola, S. (2005). Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. Journal of the Royal Statistical Society. Series A (General), 168, 307–323.
Salminen, P., Hokkanen, J., & Lahdelma, R. (1998). Comparing multicriteria methods in the context of environmental problems. European Journal of Operational Research, 104, 485–496. doi:10.1016/S0377-2217(96)00370-0.
Saltelli, A. (2007). Composite indicators between analysis and advocacy. Social Indicators Research, 81, 65–77. doi:10.1007/s11205-006-0024-9.
Saltelli, A., Tarantola, S., Campolongo, F., & Ratto, M. (2004). Sensitivity analysis in practice: A guide to assessing scientific models. New York: Wiley.
Sinha, B. K., & Shah, K. R. (2003). On some aspects of data integration techniques with environmental applications. Environmetrics, 14, 409–416. doi:10.1002/env.595.
Triantaphyllou, E. (2000). Multi-criteria decision making methods: A comparative study. Boston: Kluwer Academic Publishers.
United Nations (UN). (2001). Human development report. Oxford: Oxford University Press.
Yoon, K. P., & Hwang, C. L. (1995). Multiple attribute decision making: An introduction. Thousand Oaks, CA: Sage Publications.
Zaim, O., Fare, R., & Grosskopf, S. (2001). An economic approach to achievement and improvement indexes. Social Indicators Research, 56, 91–118. doi:10.1023/A:1011837827659.
Zanakis, S. H., Solomon, A., Wishart, N., & Dublish, S. (1998). Multi-attribute decision making: A simulation comparison of select methods. European Journal of Operational Research, 107, 507–529. doi:10.1016/S0377-2217(97)00147-1.
Zeleny, M. (1982). Multiple criteria decision making. New York: McGraw-Hill.
Zhou, P., & Ang, B. W. (2008). Indicators for assessing sustainability performance. In K. B. Misra (Ed.), Handbook of performability engineering (pp. 905–918). London: Springer-Verlag.
Zhou, P., Ang, B. W., & Poh, K. L. (2006). Comparing aggregating methods for constructing the composite environmental index: An objective measure. Ecological Economics, 59, 305–311. doi:10.1016/j.ecolecon.2005.10.018.
Zhou, P., Ang, B. W., & Poh, K. L. (2007a). A mathematical programming approach to constructing composite indicators. Ecological Economics, 62, 291–297. doi:10.1016/j.ecolecon.2006.12.020.
Zhou, P., Poh, K. L., & Ang, B. W. (2007b). A non-radial DEA approach to measuring environmental performance. European Journal of Operational Research, 178, 1–9. doi:10.1016/j.ejor.2006.04.038.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhou, P., Ang, B.W. Comparing MCDA Aggregation Methods in Constructing Composite Indicators Using the Shannon-Spearman Measure. Soc Indic Res 94, 83–96 (2009). https://doi.org/10.1007/s11205-008-9338-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11205-008-9338-0