Evaluating performance of super-efficiency models in ranking efficient decision-making units based on Monte Carlo simulations

Abstract

In response to the limitation of classical Data Envelopment Analysis (DEA) models, the super efficiency DEA models, including Andersen and Petersen (Manag Sci 39(10): 1261–1264, 1993)’s model (hereafter called AP model) and Li et al. (Eur J Oper Res 255(3): 884–892, 2016)’s cooperative-game-based model (hereafter called L–L model), have been proposed to rank efficient decision-making units (DMUs). Although both models have been widely applied in practice, there is a paucity of research examining the performance of the two models in ranking efficient DMUs. Consequently, it is unclear how close the rankings obtained by the two models are to the “true” ones. Among the very few studies, Banker et al. (Ann Oper Res 250(1): 21–35, 2017) pointed out that the ranking performance of the AP model is unsatisfactory; Li et al. (Eur J Oper Res 255(3): 884–892, 2016) and Hinojosa et al. (Exp Syst Appl 80(9): 273–283, 2017) demonstrated the L–L model’s capability of ranking efficient DMUs without addressing the ranking performance. In this study, we, thus, examine the ranking performance of the two super-efficiency models. In evaluating their performance, we carry out Monte Carlo simulations based on the well-known Cobb–Douglas production function and adopt Kendall rank correlation coefficient. Unlike Banker et al. (Ann Oper Res 250(1): 21–35, 2017), we use the rankings obtained based on the two models and the “true” ones as the basis of performance evaluation in our simulations. Moreover, we consider several types of returns to scale (RS) and study the impact of changes of some parameters on the ranking performance. In view of the importance, we also carry out additional simulations to examine the influence of technical inefficiency on the two models’ ranking performance. Based on the simulation results, we conclude: (1) Under different RS, the ranking performance of the two models remains the same when changing parameters, e.g., the distribution of input variables; (2) Under different RS, when technical inefficiency (in comparison with random noise) is more important, the two models have satisfactory performance by providing rankings that are close to, or the same as, the “true” ones; (3) The L–L model has better performance than the AP model and is more robust. This is especially true when technical inefficiency is less important; (4) Under different RS, when technical inefficiency is less important, both models have unsatisfactory ranking performance; and (5) The relative importance of technical inefficiency plays an prominent role in ranking efficient DMUs.

Appendices

Appendix 1: methods for ranking DMUs and their limitations

The cross-efficiency sorting method was first proposed by Doyle and Green (1994). In the DEA background, this method is to sort the DMUs based on a cross-efficiency matrix. The cross-efficiency matrix was originally developed by Sexton et al. (1986), who started the theme of DEA ranking. Doyle and Green (1994) pointed out that decision makers do not always have a reasonable mechanism for ranking units and, thus, recommended the cross-efficiency matrix as a hierarchical unit. The cross-efficiency ranking method uses the optimal weights evaluated by n linear programs (LPs) to calculate the efficiency score of each DMU n times. The results of all DEA cross efficiency scores can be organized as the cross-efficiency matrix. The cross-efficiency soring method allows each DMU to selfishly choose an optimal set of input and output weights and defines the average of a DMU’s efficiencies based on the optimal weights as its cross efficiency. However, cross-efficiency scores are generally not unique and depend on the alternative optimal solutions to the LPs used (Liang and Wu et al., 2008b). The efficiency obtained by this method is, thus, not unique.

The Common weight-based method is also used for ranking DMUs proposed in (Cook et al., 1990). It attempts to find a common set of weights to calculate efficiencies for all DMUs and further ranks them based on their efficiencies. Compared with the cross-efficiency method, this method evaluates the DMUs based on a common platform, i.e., the set of common weights. In this regard, the efficiency scores of units are comparable. The problem with this ranking method is that it is very difficult to find an inclusion principle to choose a common weight set. Consequently, the different principles adopted result in different common weight sets and grades between DMUs.

Nicole Adler et al. (2002) discussed the Benchmark ranking method, which sorts DMUs in two stages. In the first stage, the efficient units are ranked by simply counting the number of times they appear in the reference sets of inefficient units. The inefficient units are then ranked, in the second stage, by counting the number of DMUs that needs to be removed from the analysis before they are considered efficient. However, a complete ranking cannot be assured because many DMUs may receive the same ranking score. Another problem with this ranking method is that a DMU is highly ranked if it is chosen as a useful target for many other DMUs.

Appendix 2: Results of simulation where parameters are changed under DRS

See Figs. 15, 16, 17 and 18 and Tables 16, 17, 18, 19.

Fig. 15

Probability bar charts corresponding to the fifth set of experiments

Fig. 16

Probability bar charts corresponding to the sixth set of experiments

Fig. 17

Probability bar charts corresponding to the seventh set of experiments

Fig. 18

Probability bar corresponding to the eighth set of experiments

Table 16 Results of the fifth set of experiments (x ~ $FN~\left[ {0,2.5} \right]~$)

Table 17 Results of the sixth set of experiments ($x$ ~ $FN~\left[ {0,2.5} \right]$ and $\alpha$ ~ $FN~\left[ {1/2m,~2/m} \right]$)

Table 18 Results of the seventh set of experiments ($x$ ~ $U\left( {1,6} \right)$)

Table 19 Results of the eighth set of experiments ($x$ ~ $U\left( {1,6} \right)$ and $\alpha$ ~ $U\left( {1/3m,~3/m} \right)$)

Appendix 3: Results of simulation where parameters are changed under IRS.

See Figs. 19, 20, 21 and 22 and Tables 20, 21, 22 and 23.

Fig. 19

Probability bar charts corresponding to the ninth set of experiments

Fig. 20

Probability bar charts corresponding to the tenth set of experiments

Fig. 21

Probability bar charts corresponding to the eleventh set of experiments

Fig. 22

Probability bar corresponding to the twelfth set of experiments

Table 20 Results of the ninth set of experiments (x ~ $FN~\left[ {0,2.5} \right]~$)

Table 21 Results of the tenth set of experiments ($x$ ~ $FN~\left[ {0,2.5} \right]$ and $\alpha$ ~ $FN~\left[ {1/2m,~2/m} \right]$)

Table 22 Results of the eleventh set of experiments ($x$ ~ $U\left( {1,6} \right)$)

Table 23 Results of the twelfth set of experiments ($x$ ~ $U\left( {1,6} \right)$ and $\alpha$ ~ $U\left( {1/3m,~3/m} \right)$)