Journal of Engineering Research xxx (xxxx) xxx

Contents lists available at ScienceDirect

Journal of Engineering Research

journal homepage: www.journals.elsevier.com/journal-of-engineering-research

Full length article

A network data envelopment analysis to evaluate the performance of a

sustainable supply chain using bootstrap simulation
Masoud Vaseei a, Maryam Daneshmand-Mehr a, *, Morteza Bazrafshan a, Armin Ghane Kanafi b
a
Department of Industrial Engineering, Lahijan Branch, Islamic Azad University, Lahijan, Iran
b
Department of Mathematics, Lahijan Branch, Islamic Azad University, Lahijan, Iran

A R T I C L E I N F O A B S T R A C T

Keywords: Data envelopment analysis (DEA) is a non-parametric method to calculate the efficiency of decision-making units
Data envelopment analysis (DMU) under evaluation that perform the same activity. The frontier obtained by this method is a relative
Bootstrap frontier accessible in the real world. Due to the uncertainty of the population distribution, the accuracy of the
Network
achieved efficiency is questioned. Therefore, this research aims to present a network data envelopment analysis
Sustainable
model to evaluate the performance of a sustainable supply chain using bootstrap simulation. In this research,
using the two-step approach of data envelopment analysis and the bootstrap method, the information collected
from 25 tomato paste companies for the year 2021 has been analyzed. To illustrate the proposed method, a real
case study is considered in the Iranian tomato paste supply chain network. The findings showed that using
definitive data, 16 companies are efficient and 9 companies are inefficient, and using bootstrap simulation data,
4 companies are efficient and 21 companies are inefficient. Using the proposed framework, the overall efficiency
value has been calculated in two cases using DEA and the bootstrap model. In addition, the efficiency of the stage
is calculated separately. Based on the calculated results, if a DMU is considered efficient, its efficiency score is
equal to 1 in each of the stages. Otherwise, the cause of the inefficiency of each DMU is identified. Also, based on
the comparisons made between the proposed model and the basic models based on sensitivity analysis, the
accuracy of the proposed bootstrap-based model in introducing the number of efficient units has been better than
the basic models. Therefore, the accuracy of the used method can be concluded.

1. Introduction one [2].

There is an efficiency evaluation method called Data Envelopment
Each organization can be considered a system that produces products Analysis (DEA), which Charnes, Cooper, and Rhodes [3] were the first to
or services by receiving the resources it needs. Considering the limited introduce data envelopment analysis. Data envelopment analysis (DEA)
resources and the increasing need for products and services, each or­ is a popular non-parametric method for evaluating the relative perfor­
ganization performs better when compared to its peers when it produces mance of decision-making units (DMUs), which uses multiple inputs to
the same or more products or services by spending equal or fewer re­ produce multiple outputs [4]. Data envelopment analysis is an efficient
sources. Relative efficiency is a concept that allows the comparison of method to compare and calculate the relative efficiency of several units
the performance of any organization compared to its peers in consuming that produce similar output by spending similar inputs. But in the real
resources and producing products or services [1]. Data envelopment world, there are cases where decision-making units consist of sub-units
analysis, which was introduced by Charnes et al., (1978) based on that independently use inputs to produce outputs [5–7]. The simplest
Farrell’s study [2], is an efficient method for relative efficiency calcu­ method in facing the problem of evaluating several groups of
lation. A simple narrative of data envelopment analysis is that this decision-making units is that each group of units is considered a unit and
method is a type of optimistic evaluation that each time by selecting a the problem of comparing the efficiency of several large groups is
unit called DMU and assigning the best weights, the ratio of the reduced to the problem of comparing the efficiency of several major
weighted output to the weighted input maximizes for that DMU, while units. As can be seen in some published experiences, the problem of
this ratio is limited for all the compared DMU so that it cannot exceed comparing several groups of units is distorted to the problem of

* Corresponding author.
E-mail address: m.daneshmand@liau.ac.ir (M. Daneshmand-Mehr).

https://doi.org/10.1016/j.jer.2023.10.003
Received 18 June 2023; Received in revised form 25 September 2023; Accepted 2 October 2023
Available online 5 October 2023
2307-1877/© 2023 Published by Elsevier B.V. on behalf of Kuwait University. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).

Please cite this article as: Masoud Vaseei et al., Journal of Engineering Research, https://doi.org/10.1016/j.jer.2023.10.003
M. Vaseei et al. Journal of Engineering Research xxx (xxxx) xxx

comparing the average relative efficiency of each group that is calcu­ method. Because data coverage analysis is one of the most effective
lated separately. Today, sustainability or sustainable development has methods for evaluating the performance of institutions [19]. Because
become an important issue and an attractive area for research. Also, can this approach does not require any decision-maker to describe the
be said today, sustainability is the main issue in industrial sectors. weight values assigned to the indicators. Although the weight indices
Sustainable development is related to how to balance economic, social, are obtained directly from the data in the data coverage analysis, this
and environmental aspects in maintaining competitive strategies to method can differentiate between the figures of the criteria based on the
achieve optimal profit at the lowest price in the supply chain system. A efficiency score, as well as the inefficient resources and values of the
sustainable supply chain is the development of a conventional supply DMUs [20].
chain system that focuses on the integration of the triple line (economic, It is true that in data coverage analysis, efficiency is the criterion of
social, and environmental) in management processes to optimize the comparison, but what is used in practice to make comparisons are the
performance of each organization [8]. Integrating sustainability as a results of solving mathematical models. Mathematical models are built
core value in supply chain activities is seen by most customers as a key based on certain assumptions and their performance can be interpreted.
market opportunity to achieve competitive advantages [9]. The major Although this interpretation often becomes more difficult as the models
challenge in the current path of sustainable development affects all become more complex [21]. A criterion is suitable when it is concep­
sectors of society such as engineering and production. Manufactured tually representative of what the comparison is based on. But this is not
products must consider the impact of three aspects of sustainability enough, the criterion must be applied. It means finding a method or
throughout their life cycle. These three aspects of sustainability as an practice that can be interpreted as a comparison mechanism based on
integral part of sustainability issues are economic, environmental, and criteria. In this research, the main problem is clear and simple, with a
social factors. Based on research, it has been determined that about 80% little familiarity with data envelopment analysis, this method can
of sustainability effects are adopted in the product design phase [10]. identify several units, including the organization or any entity that re­
For example, in the manufacturing sector, the design and production of ceives things as resources and needs from the environment [22]. To
sustainable products are recognized as an important strategy to achieve compare products and services. But in the real world, there are internal
sustainability with the aim of clean production. Therefore, taking into factors that are involved in the DMU under investigation or it is
account the mentioned concepts, sustainability can be defined as the important to consider factors such as economic, social, or even envi­
ability of a supply chain to work continuously and at the same time ronmental to create simultaneous improvement [23]. In the data
provide the least environmental impact and provide economic and social envelopment analysis literature, there are published experiences that
resources for the beneficiaries. It should be noted that a sustainable either specifically address this issue or can be generalized and become
design solution effectively addresses the functional characteristics of the this issue in some way [24]. In fact, in this research, the researcher aims
product and, in addition, balances the three dimensions of sustainability to present methods or models by examining what is or can be related to
appropriately [11]. Therefore, the term sustainability in the research the subject, which, while meeting the conceptual and computational
title can be defined as the ability to do business with the long-term goal requirements, has sufficient recognition power to differentiate between
of maintaining economic, environmental, and social well-being. The units. Or, it can be equipped with such diagnostic power by proposing
supply chain is considered a collection of all relevant groups and pro­ new methods to correct the defects of multi-stage data envelopment
cesses that prepare the customer’s order. Or it can be defined as all ac­ analysis models and its use for evaluating sustainable supply chain
tivities related to the transfer and flow of goods and services, including management and making improvements at the same time [25]. To
the flow of relevant information from the source of materials to the end calculate the real efficiency, we need to know the maximum output that
user [12]. Therefore, sustainable supply chain management is used to can be produced from an input. In other words, we need a target func­
simultaneously improve economic, social, and environmental perfor­ tion called the production function, the value of the function for each
mance. Also, it provides a competitive advantage for companies because input represents the maximum amount that can be produced from it.
the opportunity to use natural resources fairly, be careful not to destroy Unfortunately, in the real world, we are unable to calculate the
the environment, and be responsible for the health and safety of em­ maximum output from a given input [26]. In other words, the true ef­
ployees and economic work makes us stand out from other competitors ficiency frontier is unclear. To overcome this problem and provide the
[13]. efficiency value of each unit, a function can be specified as a production
The supply chain includes all parts that are directly or indirectly function. In other words, consider an estimate of the true boundary
related to customer demands. In addition, due to the expansion and instead. This work can be done in two parametric and non-parametric
global shape of markets, supply chain management (SCM) has become ways. In parametric models, a specific functional form with some un­
necessary for industries [14]. Supply chain management is one of the known parameters is considered for efficiency. In this method, the aim is
important tools for companies to maintain competition and optimize to determine the unknown parameters with the help of the available
operational processes. Today, supply chain management is facing sample data. One of the biggest disadvantages of the parametric method
increasing challenges. On the one hand, with rapid economic develop­ is imposing a specific functional form on the model. In addition, the
ment, shopping behavior has increased significantly, which produces a units are evaluated against a boundary that is usually unreachable in the
large number of pollutants [15]. Governments around the world are real world. But in the non-parametric method, no specific functional
imposing strict sustainability requirements on commodity production, form is imposed on the model. And the sample data is allowed to
which increase the environmental pressure on supply chain companies determine the shape of the production function itself. In this method,
[16]. Supply chain management (SCM) integrates supply and demand instead of the units being evaluated with a parametric inaccessible
management inside and outside companies. Sustainable Supply Chain boundary, they are evaluated with an accessible boundary made of data
Management (SSCM) is a critical guide to the principles and practices of [2]. Considering that the existence of a performance evaluation system
sustainable management and total supply chain management. Sustain­ in different dimensions of organizations and in general in
ability factors force supply chain management to focus not only on decision-making units is very important and the lack of a suitable per­
economic criteria but also on social and environmental criteria [17]. In formance evaluation system to evaluate the performance of
other words, organizations try to combine values with the management decision-making units in the use of available resources and facilities will
and improvement of environmental, social, and economic factors have negative consequences for them. Different organizations must
throughout the sustainable supply chain [18]. Sustainable supply chain develop an efficient evaluation system to evaluate and improve their
management has attracted the attention of many researchers in the past performance. Today, competition, with the fading of geographical bor­
years. Among the evaluation methods of sustainable supply chain ders, the expansion of information and communication technologies, the
management, data envelopment analysis is a suitable evaluation globalization of new dimensions, and efforts to improve productivity,

2
M. Vaseei et al. Journal of Engineering Research xxx (xxxx) xxx

form the main basis of this competition. Examining the performance of numerical simulation of real data, and calculation of the efficiency of
countries that have seen significant economic growth in recent years simulated data are provided. In the fourth section, a practical example is
indicates that the main source of increasing their economic growth was considered. Finally, in the fifth section, a conclusion and suggestions for
mainly increasing efficiency and productivity [17]. future research are presented.
Data envelopment analysis is a non-parametric method for calcu­
lating the efficiency of a group of units that perform the same activity. 2. Literature review
The border obtained from this method is a relative border accessible in
the real world [18]. As opposed to the parameter method in which the In recent years, managers and decision-makers have not only focused
units are measured relative to a boundary that is generally inaccessible on improving internal processes but also considered sustainability fac­
in the real world. The efficiency obtained from this method is a relative tors in supply chains [27]. Kabadormos and Erdoğan [28] stated that
value and not a real value. In other words, the efficiency obtained from sustainability factors include three economic, social, and environmental
this method is an estimate of the actual value of efficiency. Due to the aspects. For the first time, Fukuyama and Weber [29] put forward this
uncertainty of the population distribution, the accuracy of the estimated argument that, considering that in normal data envelopment analysis
efficiency is questioned [19]. In other words, the only information we models, the production process is considered a black box, and the in­
have about society is a sample of data. It is best to define a boundary as teractions between inputs and outputs and Intermediate factors are not
efficiency and evaluate units against it. The efficiency values obtained considered. Therefore, intermediate values should be included in the
from the comparison with this border depend on the degree of structure of the network of decision-making units. Mirhedayatian et al.
compatibility and the degree of skewness of this border from the main [30], by using a data envelopment analysis model, proposed the eval­
border. As much as this frontier is a correct estimate of the original ef­ uation of the green supply chain in the presence of dual-role factors,
ficiency frontier, the resulting efficiency values are as close to the undesirable outputs, and fuzzy data. Ding [31] considered four influ­
original efficiency values. Therefore, providing a correct estimate of the encing factors of financial return, energy consumption, benefits, and
main parameter depends not only on the original structure of the esti­ environmental effects to measure the benefit-cost criterion. Based on
mator but also on the sample under investigation. How much informa­ this criterion, the total costs and benefits of the projects were measured
tion a sample can provide to the estimator should be discussed. What is and compared. Recently, researchers have begun to investigate
clear is the small number of samples observed, and in other words, the two-stage structures in supply chain networks, the results of the first
small size of the sample for correct estimation. Therefore, we should stage of which become the input of the second stage [7]. To evaluate the
look for a method to generate data from the unknown population to performance of such networks, researchers use conventional coverage
solve the problem of lack of samples [20]. Bootstrap, presented and analysis models. For example, Shiu and Chen [32] have used data
developed by Efron, is based on simulating the data generation process. envelopment, analysis models. Fukuyama and Mirdehghan [33] pro­
A big benefit of Bootstrap is its simplicity. This method is used to esti­ posed a two-step method to identify the efficiency status of each
mate the standard error and confidence interval for complex estimators decision-making unit. In their article, Tajbakhsh and Hosseini [34]
of distribution parameters, such as percentile points, ratios, odds ratios, focused on evaluating supply chain operations that maximize economic
and right-sided correlation coefficients [21]. In addition, it is a suitable returns and minimize environmental impacts to meet social expecta­
method to control and check the stability of the results. Therefore, this tions. For this purpose, a multi-stage data envelopment analysis model
research seeks to try to use the bootstrap method with data generation, has been developed, which is used to evaluate the sustainability of a
keeping in mind the accuracy of efficiency using the data envelopment supply chain with several business partners. The focus of this article is on
analysis method in evaluating the performance of supply chain sus­ using the capabilities of data envelopment analysis to comprehensively
tainability, and the main goal of this research is to provide a multi-level evaluate the performance of sustainable supply chains. Khodakarami
model [22]. It is a one-step network data coverage analysis that can et al. [35] showed one of the applications of data envelopment analysis,
evaluate the efficiency of the entire system and its subsystems by which is the evaluation of two-stage processes in which all the outputs of
applying stability factors in the supply chain at once, taking into account the first stage are intermediate values and are considered as the inputs of
the intermediate factors of different parts of this network [23]. Calculate the second stage have paid The resulting two-stage coverage analysis
the bootstrap simulation method so that it can be used in case studies. models to evaluate both the overall efficiency score of the entire process
Therefore, it can be stated that the main question of this research is as and each of the stages separately. Simonov et al. [36] have presented a
follows: methodology for evaluating a sustainable supply chain. Chen et al. [37],
sustainable product design is considered one of the most important
• How to evaluate the sustainability performance of a supply chain measures to achieve sustainability in this article. In this article, it is
using the bootstrap method and data envelopment analysis? proposed to improve the environmental performance of a product
Thus, the main objective of the current study is as follows: through product design, using a two-stage data envelopment analysis
• Evaluation of sustainability performance in a supply chain using a (DEA) network to evaluate the performance of sustainable product
combined method of bootstrap and data envelopment analysis. design. The design efficiency criterion is conceptualized as a key mea­
To evaluate the performance through the coverage analysis surement of design performance in terms of how to combine several
calculation, it is necessary to accurately estimate the efficiency product specifications and features in a product design that leads to
frontier. It is clear that the better the estimated boundary, the closer reduced environmental impacts or better environmental performance.
the obtained efficiency values will be to the actual efficiency. A main Therefore, a two-stage network DEA model is built to evaluate sustain­
unique contribution of the paper is as follows: able design performance with an industrial design module and a bio­
• Explain sustainability concepts to calculate efficiency score using logical design module. To demonstrate the applications of the
DEA. DEA-based method, key engineering data, product characteristics, and
• Development of a hybrid model using the DEA-Network structure emission performance in the vehicle emissions test database published
based on Bootstrap to explain the sustainability pattern in the tomato by the US EPA are used to evaluate the sustainable design performance
paste industry in Iran. of various vehicle manufacturers. The test results of this article show
that sustainable design does not need to mean a compromise between
In the following, this paper will be organized as indicated. In the traditional and environmental characteristics. By addressing the inter­
second section, a literature review is presented, and in the third section, relationship of subsystems in product design, a company can find the
the research method that includes the method of estimating perfor­ most efficient way to combine product specifications and features that
mance and sustainable products through bootstrap simulation, lead to reduced environmental impacts or better environmental

3
M. Vaseei et al. Journal of Engineering Research xxx (xxxx) xxx

performance. This paper contributes to the existing literature by not have any of the problems of the previous methods.
providing a new research framework for evaluating sustainable design Therefore, in this research, we are faced with the problem of esti­
performance and also by proposing an innovative program of a mating the efficiency frontier; therefore, we must first get an estimate of
two-stage DEA network to find the most environmentally efficient the efficiency frontier, then compare the efficiency of each unit with
method to achieve better environmental performance through product respect to that limit. It is clear that the better the estimated boundary is,
design. Krmac and Boban [38] have presented a model based on data the closer the obtained efficiency score is to the actual efficiency value.
envelopment analysis to evaluate the supply chain and evaluate the ef­ Most researchers generally use data envelopment analysis or super-
ficiency of suppliers. They evaluated the environmental efficiency of efficiency model of data envelopment analysis in efficiency studies.
suppliers by using data envelopment analysis models. Pachar et al. [39] The technical efficiency calculated by the traditional CCR-DEA model
presented a performance evaluation model based on the two-stage has inherent disadvantages such as the overestimation of efficiency due
method of data envelopment analysis to measure the joint impact of to random factors. In addition, relative efficiency does not mean abso­
sustainable operations and operational activities on the business per­ lute efficiency; therefore, in this research, to cover this inherent defect of
formance of a retail company. A case study of an Indian e-retail chain is the data envelopment analysis models from bootstrap-based data
presented to reveal the appropriateness of the proposed models. Shirazi envelopment analysis models and multi-stage data envelopment anal­
and Mohammadi [40] presented a DEA model to evaluate airline effi­ ysis models to measure the efficiency of units Used under evaluation. In
ciency in the presence of adverse exits. They considered DEA as a fact, in this research, we want to investigate the effect of stability factors
mathematical modeling technique. Vafaei et al. [41] in research entitled on the performance score of multi-stage data envelopment analysis
"Designing a sustainable multi-channel supply chain distribution models and bootstrap-based data envelopment analysis models. In fact,
network" stated that the existence of numerous social, economic, and we want to check how the bootstrap method can be used to estimate the
environmental problems and the need to improve long-term perfor­ reliable efficiency frontier in order to show its superiority over other
mance in organizations make supply chain plans towards sustainable conventional and traditional methods.
plans has pushed On the other hand, due to the emergence of new According to the review of numerous articles, most of the studies
technologies and the creation of new business models, increasing the related to sustainability are focused on supply chain management. In
level of competition among retailers and increasing customer expecta­ Table 1, the main indicators and sub-indices for sustainability have been
tions, the use of multi-channel distribution systems is growing today. examined separately.
Therefore, the appropriate distribution channel according to the type of
product and the number of vehicles for transporting products is an 3. Research method
important issue in retail systems. Tavassoli et al. [42] presented a data
envelopment analysis model and fuzzy data envelopment analysis for The main issue in current research is the design of a network data
evaluating sustainable supply chains. The proposed model evaluates the
optimistic and pessimistic sustainability of supply chains. The alpha
approach is used to solve the proposed models. This paper aims to Table 1
develop a two-level data envelopment analysis model in the presence of Classification of main and secondary indicators for supply chain sustainability.
undesirable outputs. Dey et al. [43] presented a data envelopment Main indicator Sub indicator Description References
analysis model for managing the sustainable supply chain performance Economic Return on investment Recovery of invested [41,42]
of small and medium enterprises. Zhao et al. [44] stated in their research products financial resources in a
that performance or efficiency evaluation is very important for effective stable situation through
supply chain management, especially in the field of sustainable devel­ various activities such as
reuse, recycling, and sale
opment. This paper uses the two-stage envelopment analysis, which has
of waste and waste
been widely used to evaluate supply chain efficiency. The method pro­ materials.
posed in this paper has been validated through a numerical example of a Investing in research Some financial resources [43,44]
sustainable supply chain of producers and suppliers in resin companies. and development for are invested to support
sustainable products the production of
Moghaddas et al. [45] proposed an assessment approach based on the
sustainable products.
network data envelopment analysis (DEA) to choose an efficient strategy Sustainable product The organization’s efforts [45,46]
for each stage of a sustainable supply chain (SSC) network. This design to reduce to design products to
approach seeks to provide a sustainable design with decision-making to consumables and cost reduce consumables and
avoid imposing additional costs on SCs that result from noncompliance costs.
The ability of green Ability to organize [47,48]
with environmental and social issues. Chen et al. [46] presented a
logistics packaging, labeling, and
regret-based cross-efficiency (CE) evaluation method in a general transportation of
two-stage DEA system. For this purpose, introduced regret theory to products in an
portray DMs’ psychological preferences and investigates two-stage CE environmentally friendly
evaluation models based on regret theory. manner.
Environmental Product design to Designing the [43,49]
reduce the impact on organization’s product to
2.1. Research gap the environment reduce environmental
impact, including using
Several decision-making units that are evaluated in DEA models have environmentally friendly
materials for products for
an efficiency score of one, so there is no distinction between them in easier end-of-life disposal
theory. Most of the decision-makers are looking for a complete ranking Quick response to The organization’s quick [47,50]
of DMUs so that they can better evaluate their performance and move customers and response to customers
towards further improvement. Until now, various methods have been market demand for and market demand,
sustainable products awareness of the benefits
proposed for ranking efficient and inefficient units and many articles
of using environmentally
have been published in this field. Each of these methods has some friendly and green
problems and they are not able to rank all DMUs. The bootstrap method products
presented in this article can rank all the units. The bootstrap process uses Social Values and beliefs of the benefit of society is [51,52]
computing power instead of theoretical analysis. The bootstrap perfor­ organizations or more than individual
individuals in which benefits.
mance scores implicitly use linear programming. For this reason, it does

4
M. Vaseei et al. Journal of Engineering Research xxx (xxxx) xxx

envelopment analysis model to evaluate the performance of the sus­

ur ≥ 0; vi ≥ 0
tainable supply chain using bootstrap simulation. In this research, due to
the uncertainty of the population distribution, the accuracy of the ach­ The fractional programming model of Eq. (1) is known as the frac­
ieved efficiency is questioned. Hence, the bootstrap simulation process tional CCR model. Where ur is the weight of the rth outputs; vi is the ith
has been used to increase the accuracy of the achieved performance. To weight of inputs. yro and xio are respectively the values of rth outputs and
illustrate the proposed method, a real case study in the sustainable ith inputs for the DMU under evaluation. Also, yrj and xij are respectively
supply chain network of tomato paste is considered. This research fol­ the values of the rth outputs and the value of the ith input for the jth
lows two main approaches, which include: 1- the data envelopment DMU. S is several outputs; m is the number of inputs and n represents the
analysis approach and 2- the bootstrap simulator approach, which we number of DMUs. Note that the definition of efficiency in the fractional
will discuss in brief below. CCR model is "the result of dividing the weighted composition of outputs
by the weighted composition of inputs". Then, using the model of Eq. (2),
3.1. Preliminary definitions the non-linear fractional CCR model is converted into a linear model as
∑
follows. To linearize the equation m i=1 vi xio is assumed to be t which is
1

3.1.1. Data envelopment analysis (DEA) shown in Eq. (2).

Data Envelopment Analysis (DEA) is one of the most efficient and ∑
maxz = tur yro (2)
common performance evaluation techniques with objective and exper­
imental documentation and a quantitative method. DEA is a non-
subject to :
parametric technique to measure and evaluate the relative efficiency
of a set of phenomena (organizations) with definite inputs and outputs ∑
tVi xio = 1
[53]. One of the important advantages of this technique is that it does
not need the assumptions and limitations of traditional performance ∑ ∑
measurement models. Since the introduction of this method, it has been ur yrj − vi xio ≤ 0
widely used in most organizations, both for-profit and non-profit. The
basis of using this method is to create a consistent ratio of the weighted ur ≥ 0; vi ≥ 0
sum of outputs to the weighted sum of inputs for each decision-making By changing the variable tur = ur andtvi = vi , the final linearized
unit (DMU) [54]. Data envelopment analysis has unique features that model is written according to Eq. (3).
distinguish it from other classical and parametric models of performance ∑
measurement. Therefore, the main advantages of the DEA method are as maxz = tur yro (3)
follows [55]:
subject to :
• Realistic assessment ∑
• Simultaneous evaluation of the set of factors vi xio = 1
• No need for predetermined weights
∑ ∑
• Compensatory ur yrj − Vi xio ≤ 0
• Assessment with borderline orientation instead of central orientation
• Visualizing the best functional state rather than the desired state ur ; vi ≥ 0

3.1.2. Charnes, Cooper, and Rhodes model (CCR) 3.1.3. Bootstrap method in data envelopment analysis approach
One of the first models is called data envelopment analysis (CCR) To estimate the standard deviation and confidence interval for the
which was developed by Charnes et al. [3] to calculate efficiency scores mean, median, ratio, and other distribution parameters, the "bootstrap"
with multiple inputs and outputs. The basis of this model is the defini­ method is used. Of course, this method can be used to estimate the error
tion of efficiency as the ratio of one output to one input. In other words, of the regression coefficients and also to perform the hypothesis test.
in the CCR model, to calculate technical efficiency, instead of using the When the sample size is small and the accuracy of the estimators is
ratio of one output to one input, the ratio of the weighted sum of outputs important, the bootstrap method can calculate the error by the resam­
(virtual output) to the weighted sum of inputs (virtual input) is used. pling method and provide the confidence interval or standard deviation
One of the most important topics in data coverage analysis is to deter­ suitable and versatile. Also, this method is used to estimate the variance
mine the type of return to scale. Returns to scale show the relationship of the estimators when the probability distribution function of the es­
between changes in the inputs and outputs of the system. One of the timators is uncertain or complex [56]. Considering that it is not possible
abilities of the DEA method is to determine the efficiency at different to obtain information about the input and output variables of all
scales in the system units. In the CCR model, to determine the highest decision-making units to calculate the efficiency, it is not possible to
efficiency ratio and to involve the number of inputs and outputs of other calculate the parameters related to the statistical population of these
decision-making units in determining the optimal weights for the unit units and it can only be done by taking a small sample from this sta­
under review, the basic model of Eq. (1) was proposed: tistical population investigated the characteristics of this sample. In the
∑
s data envelopment analysis method, a small sample of decision-making
ur yro units is examined, and the efficiency of the decision-making units is
maxz = r=1
∑
m (1) calculated by building an efficiency frontier based on the limited in­
vi xio formation related to the units in the sample. It is obvious that the more
i=1
the marginal efficiency estimated from the sample is more skewed than
subject to : the main marginal efficiency related to the statistical population, the
validity of the estimated marginal efficiency will be lower and the more
∑
s
ur yrj this marginal efficiency matches the original marginal efficiency, the
r=1
≤ 1j = 1.2.….n calculated efficiency values will be equal to the values The main per­
∑ formance will be closer. Therefore, to solve this important limitation
m
vi xij
i=1 (lack of knowledge about the statistical population), a method should be
used that can provide a reliable estimate of the information related to

5
M. Vaseei et al. Journal of Engineering Research xxx (xxxx) xxx

the main statistical population from the limited sample that is available.
The bootstrap method is a method that tries to solve this limitation by
using simulation and a data lead process. The bootstrap method, by
using a special algorithm, simulates several new samples based on the
information related to the input and output variables of the units in the
sample, and with this, for each of the units in the original sample, we
will have several simulated units, corresponding to the same original
unit. Then, the efficiency of the units in each of the simulated samples is
calculated using the data envelopment analysis method. At this stage, by Fig. (2). How the bootstrap simulation method works [35].
calculating the average efficiency of each of the simulated units corre­
sponding to the original units and calculating the difference between – Step 1: For each DMU that has xk inputs and yk outputs, we obtain the
this average and the efficiency obtained for the units in the original efficiency score ̂
θk by using a suitable DEA model according to the
sample, the degree of skewness of the results of the data envelopment nature and main structural form of the system.
analysis method can be obtained. [90]. Fig. 1 shows the model execution
– Step 2: Using the value of ̂ θk for each DMU using bootstrap, the
process in the framework.
random efficiency score θ∗1b , …, θ∗nb is calculated.
– Step 3: We calculate the simulation value for (Xkb , Ykb ) according to
3.2. Solution approach
Eq. (4):
( / ∗) ( / ∗)
The main idea of the estimation method is to evaluate performance Xkb = ̂ θk θnb .Xk , Ykb = ̂θk θnb .Yk (4)
and sustainable products through bootstrap simulation, numerical
simulation of real data, and calculation of the efficiency of simulated
data. In Fig. (2) the performance of the method is briefly shown. For the – Step 4: For each simulated sample, we must again use the DEA
details of the proposed model, we consider the following steps. method to calculate the efficiency scoreθ∗bk .

By repeating the second to fourth steps to B times, we will have a set

Fig. 1. Research framework.

6
M. Vaseei et al. Journal of Engineering Research xxx (xxxx) xxx

of estimated scoreθ∗bk . Table 2

Data set.
4. Results DMU Supply chain Inputs Middle Outputs

X1 X2 Z12 Y21 Y22

In this section, a practical example is considered for further analysis
1 Oila 0.83 0.59 0.62 0.83 0.79
and investigation of the introduced approach. For this purpose, an Ira­
2 Daland 0.75 0.73 0.85 0.86 0.62
nian supply chain of tomato paste has been considered. For this purpose, 3 Sahar 0.82 0.74 0.45 0.45 0.96
25 tomato paste production companies have been evaluated. Table 6 4 Kambiz 0.92 0.89 0.68 0.83 0.99
shows the factors and symbols used. The structure of the considered 5 Mohsen 0.84 0.74 0.74 0.73 0.96
supply chain is also shown in Fig. 3. In this example, we are dealing with 6 Orom Ada 0.87 0.57 0.91 1 0.95
7 Rojin 0.73 1 0.68 0.58 0.89
multiple supply chains. Stage (1) is the manufacture and stage (2) is the
8 Mahram 0.94 0.67 1 0.64 0.94
supplier. In the input of the supplier stage, the cost of purchasing ma­ 9 Chin Chin 1 0.76 0.61 0.86 1
terials is considered an economic factor, and the welfare cost of em­ 10 Tabarok 0.74 0.91 0.66 0.66 0.79
ployees is considered a social factor. In Fig. 3, we show the input vector 11 Kamalan 0.62 0.45 0.52 0.83 0.69
12 Sorkh Abi 0.65 0.87 0.65 0.86 0.52
that enters the first stage withx2i . z12 is the output vector that leaves step
13 Zoshk 0.67 0.89 0.65 0.68 0.86
(1) and enters step (2) as an input vector. This variable is known as the 14 1&1 0.97 0.78 0.78 0.63 0.89
intermediate variable. Air pollution variables are shown as adverse 15 Nazchin 0.83 0.64 0.74 0.78 0.45
output and environmental index through Y22 in the second step. Y21 , is 16 Ata 0.75 0.56 0.71 0.68 0.75
17 Khoshab 0.89 0.78 0.78 0.78 0.59
the final output of the system. That is the number of supplies supplied.
18 Bijan 0.69 0.56 0.78 0.74 0.74
The data is shown in Table (2). 19 Yara 0.78 0.76 0.81 0.66 0.65
Based on the results obtained from the overall and staged efficiency 20 Majid 0.79 0.81 0.56 0.66 0.59
in this supply chain shown in Table 7, DMU1, DMU2, DMU3, DMU6, 21 Ghods 0.68 0.49 0.62 0.73 0.79
DMU7, DMU8, DMU9, DMU11, DMU12, DMU15, DMU16, DMU17, 22 Tara 0.86 0.83 0.65 0.76 0.52
23 Tabiat 0.78 0.84 0.65 0.58 0.76
DMU18, DMU21, DMU22, and DMU24 are the overall performance of
24 Delpazir 0.69 0.79 0.78 0.63 0.89
the network. Because a DMU like K is called efficient if and only if the 25 Taravat 0.84 0.64 0.64 0.58 0.56
objective function θ for model 1 has an optimal solution equal to one.
Otherwise, it is inefficient, considering that among the known efficient
units, there are units that are inefficient in the first or second stage, but
Table 3
the model has identified them as generally efficient. Concerns are felt Overall efficiency and stage efficiency.
regarding the identification of fully efficient units. Because we argue
DMU Overall Stage 1 Stage 2 Bootstrap
that due to the inefficiency of some DMUs in the first and second stages
efficiency ̂ efficiency efficiency efficiency
of the network, these units are not considered efficient. For example,
θk
̂
θ k1 ̂
θ k2 θ∗nb
DMU1, which is efficient in the first stage but inefficient in the second
1 1 1 0.82 0.91
stage, cannot be included in the set of efficient DMUs. The reason is that
2 1 1 1 1
several decision-making units in the second stage did not perform well 3 1 1 0.78 0.89
compared to the first stage and were identified as inefficient units, and 4 0.90 0.9 0.72 0.81
this inefficiency has caused the overall inefficiency of the decision- 5 0.82 0.82 0.72 0.77
making unit. Therefore, by using the average efficiency of the first and 6 1 1 1 1
7 1 0.90 0.90 0.90
second steps, we consider an efficiency value for the bootstrap value to 8 1 0.61 1 0.80
obtain the value of inputs and outputs according to the third step of the 9 1 1 0.64 0.82
proposed method. The bootstrap efficiency value obtained from the 10 0.93 0.92 0.88 0.90
average efficiency of the first and second stages is shown in the last 11 1 1 1 1
12 1 1 1 1
column of Table 3 with the symbolθ∗nb .
13 0.98 0.94 0.97 0.95
Using the calculated bootstrap efficiency, we calculate the bootstrap 14 0.78 0.78 0.97 0.87
random input and output based on Table 4 to calculate the overall and 15 1 1 0.84 0.92
final efficiency of the DMUs. 16 1 0.87 0.93 0.90
Taking into consideration the random data obtained based on 17 1 1 0.78 0.89
18 1 0.79 1 0.89
Bootstrap, the overall and staged efficiency of the supply chain has been 19 0.96 0.64 0.92 0.78
recalculated. According to the results shown in Table 5, in this case, only 20 0.93 0.93 0.80 0.86
the overall efficient units are known which were efficient in both stages. 21 1 1 0.99 0.99
Therefore, only DMU2, DMU6, DMU11 and DMU12 are considered 22 1 1 0.76 0.88
23 0.94 0.94 0.84 0.89
effective.
24 1 0.78 1 0.89
25 0.88 0.81 0.78 0.79

4.1. Sensitivity analysis

According to the data envelopment analysis model considered in the

set of 3 equations, by defining the variables θ and λ as dual variables, the
dual model becomes a set of Eq. (5).
minθ (5)

subject to

λj xij ≤ θxio
Fig. 3. The structure of the Iranian tomato paste supply chain.

7
M. Vaseei et al. Journal of Engineering Research xxx (xxxx) xxx

Table 4 Table 6
Random data-based bootstrap. Changing deterministic input.
DMU Supply chain Inputs Middle Outputs DMU Proposed model Deterministic Maximum Δxi
efficiency input value changing of
X1 X2 Z12 Y21 Y22
input
1 Oila 0.91 0.54 0.33 0.28 0.22 θxio
2 Daland 0.75 0.73 0.85 0.86 0.62
X1 X2 X1 X2 X1 X2
3 Sahar 0.92 0.83 0.37 0.37 0.36
4 Kambiz 1.03 0.92 0.63 0.52 0.51 4 0.90 0.92 0.89 0.82 0.80 0.10 0.09
5 Mohsen 0.93 0.69 0.69 0.50 0.48 5 0.82 0.4 0.74 0.68 0.60 0.16 0.14
6 Orom Ada 0.87 0.57 0.91 1 0.95 10 0.93 0.74 0.91 0.68 0.84 0.06 0.07
7 Rojin 0.81 1.11 0.75 0.28 0.25 13 0.98 0.67 0.89 0.65 0.87 0.02 0.02
8 Mahram 1.75 0.79 0.78 0.8 0.32 14 0.78 0.97 0.78 0.75 0.60 0.22 0.18
9 Chin Chin 1 0.76 0.61 0.86 1 19 0.96 0.78 0.76 0.74 0.72 0.04 0.04
10 Tabarok 0.65 0.80 0.58 0.42 0.69 20 0.93 0.79 0.81 0.73 0.75 0.06 0.06
11 Kamalan 0.62 0.45 0.52 0.83 0.69 23 0.94 0.78 0.84 0.73 0.78 0.05 0.06
12 Sorkh Abi 0.65 0.87 0.65 0.86 0.52 25 0.88 0.84 0.64 0.73 0.56 0.11 0.08
13 Zoshk 0.55 0.84 0.17 0.69 0.73
14 1&1 0.78 0.98 0.32 0.56 0.46
15 Nazchin 0.69 0.58 0.39 0.79 0.62
16 Ata 0.85 0.65 0.47 0.58 0.46 Table 7
17 Khoshab 0.79 0.74 0.68 0.39 0.49 Changing bootstrap input.
18 Bijan 0.46 0.68 0.49 0.69 0.46
DMU Proposed model Bootstrap Maximum Δxi
19 Yara 0.46 0.39 0.74 0.62 0.39
efficiency input value changing of
20 Majid 0.49 0.71 0.49 0.39 0.78
input
21 Ghods 0.49 0.39 0.48 0.69 0.48
θxio
22 Tara 0.79 0.78 0.55 0.63 0.42
23 Tabiat 0.65 0.74 0.55 0.69 0.46 X1 X2 X1 X2 X1 X2
24 Delpazir 0.59 0.89 0.56 0.36 0.79
1 0.89 0.91 0.54 0.80 0.48 0.11 0.06
25 Taravat 0.25 0.35 0.79 0.69 0.48
3 0.91 0.92 0.83 0.83 0.75 0.09 0.08
4 0.90 1.03 0.92 0.92 0.82 0.11 0.10
5 0.82 0.93 0.69 0.76 0.56 0.17 0.13
Table 5 7 0.77 0.81 1.11 0.62 0.85 0.19 0.26
8 0.78 1.75 0.79 1.36 0.61 0.39 0.18
Overall and stage bootstrap efficiency.
9 0.75 1 0.76 0.75 0.57 0.25 0.19
DMU Overall efficiency ̂
θk Stage 1 efficiency Stage 2 efficiency 10 0.93 0.65 0.80 0.60 0.74 0.05 0.06
̂
θ k1 ̂
θ k2 13 0.77 0.55 0.84 0.38 0.64 0.17 0.20
14 0.78 0.78 0.98 0.60 0.76 0.18 0.22
1 0.89 1 0.82 15 0.65 0.69 0.58 0.44 0.37 0.25 0.21
2 1 1 1 16 0.55 0.85 0.65 0.46 0.35 0.39 0.30
3 0.91 1 0.78 17 0.68 0.79 0.74 0.53 0.50 0.26 0.24
4 0.90 0.90 0.72 18 0.64 0.46 0.68 0.29 0.43 0.17 0.25
5 0.82 0.82 0.72 19 0.96 0.46 0.39 0.44 0.37 0.02 0.02
6 1 1 1 20 0.93 0.49 0.71 0.45 0.66 0.04 0.05
7 0.77 0.90 0.90 21 0.82 0.49 0.39 0.40 0.31 0.09 0.08
8 0.78 0.61 1 22 0.74 0.79 0.78 0.58 0.57 0.21 0.21
9 0.75 1 0.64 23 0.94 0.65 0.74 0.58 0.69 0.07 0.05
10 0.93 0.92 0.88 24 0.89 0.59 0.89 0.52 0.79 0.07 0.10
11 1 1 1 25 0.88 0.25 0.35 0.22 0.31 0.03 0.04
12 1 1 1
13 0.77 0.94 0.97
14 0.78 0.78 0.97
inefficient DMUs values and bootstrap simulated values, respectively.
15 0.65 1 0.84
16 0.55 0.87 0.93 Figs. 4 and 5 show the amount of deterministic and bootstrap inputs
17 0.68 1 0.78 along with possible changes in the inputs. Based on Figs. 4 and 5, it can
18 0.64 0.79 1 be seen that the amount of changes in the input has been reduced as
19 0.96 0.64 0.92 much as possible in both deterministic and bootstrap situations.
20 0.93 0.93 0.80
21 0.82 1 0.99
The inputs under the influence of this research are deterministic
22 0.74 1 0.76 inputs and the bootstrap simulator. Considering that the maximum
23 0.94 0.94 0.84 allowed changes in the inputs isθxio , therefore, by multiplying the in­
24 0.89 0.78 1 efficiency value in the input of each of the units under evaluation, the
25 0.88 0.81 0.78
value of the input changes is obtained. For example, if Kambiz Company
reduces the input X1 by 0.1 units and the X2 input by 0.09 units, it can
reach the level of efficiency on the efficient border. And also, in the
λj yrj ≥ yrj
changes of bootstrap simulation, Kambiz Company can reduce the
amount of X1 input by 0.11 units and X input by 0.1 units, it can be
λj ≥ 0
placed on the efficient frontier. Fig. 6 shows the changes in efficiency
The set of Eq. (5), which is known as the cover form of the CCR scores of Iranian tomato paste manufacturing companies based on input
model, θxio is the maximum amount of savings in inputs that can be changes based on deterministic inputs and bootstrapped simulation.
placed on the efficiency frontier of the inefficient unit as an efficient unit According to the observed changes based on the deterministic and
by reducing the number of inputs. By determining the total efficiency bootstrapped efficiency score, it can be seen that the efficiency value in
value for the DEA model and the bootstrapped simulation values in the the bootstrapped mode is always lower than the deterministic mode.
previous step, the value of θxio is calculated for all the inputs in both Therefore, the results of bootstrap simulation to estimate the efficiency
considered modes to the maximum amount of change in the inputs to be frontier overcome the weakness of over-estimation of data envelopment
on determining the efficiency frontier.Tables 6 and 7 are calculated for analysis models.

8
M. Vaseei et al. Journal of Engineering Research xxx (xxxx) xxx

Fig. 4. Input changing of the deterministic value.

Fig. 5. Input changing of the bootstrap value.

Fig. 6. Changing of the deterministic and bootstrap value.

9
M. Vaseei et al. Journal of Engineering Research xxx (xxxx) xxx

Table 8 • In this research, a management method for measuring overall and

Comparison between bootstrap, CCR and BCC models. staged relative efficiency has been presented using data envelopment
DMU Bootstrap CCR BCC analysis and bootstrap simulation.
efficiency efficiency efficiency • By using the presented framework, managers are allowed to measure
1 0.89 1 0.94 system weaknesses through efficiency calculations.
2 1 1 1 • Calculation of efficiency in this presented method is done so that
3 0.91 1 1 efficiency is available as a quantity.
4 0.90 1 0.82
5 0.82 0.92 0.82
6 1 1 1
6. Discussion and conclusion
7 0.77 1 0.90
8 0.78 0.71 0.64 Data envelopment analysis is a non-parametric method to calculate
9 0.75 1 1 the efficiency of units under evaluation that perform the same activity.
10 0.93 1 0.88
The boundary obtained by this method is a relative boundary accessible
11 1 1 1
12 1 1 1 in the real world. As opposed to the parameter method in which units are
13 0.77 1 1 measured relative to a boundary, which is generally unavailable in the
14 0.78 0.88 0.82 real world. The efficiency obtained from this method is a relative value
15 0.65 1 1 and not a real value. In other words, the efficiency obtained from this
16 0.55 0.97 0.93
17 0.68 1 0.78
method is an estimate of the efficiency value. Due to the uncertainty of
18 0.64 0.89 0.66 the population distribution, the accuracy of the achieved efficiency is
19 0.96 0.74 0.62 questioned. Data envelopment analysis is a powerful tool for calculating
20 0.93 1 0.80 efficiency. However, the sensitivity of its models to sample change re­
21 0.82 1 0.99
duces the confidence in the efficiency values obtained from this method.
22 0.74 1 0.76
23 0.94 0.94 0.84 Sustainable supply chain management has attracted the attention of
24 0.89 0.78 0.64 many researchers in the past years. Among the evaluation methods of
25 0.88 0.81 0.78 sustainable supply chain management, data envelopment analysis is a
Total number of efficient 4 16 8 suitable evaluation method. Because data coverage analysis is one of the
DMU
most effective methods for evaluating the performance of institutions.
Because this approach does not require any decision-maker to describe
Also, Table 8 shows the performance results of DMUs according to the weight values assigned to the indicators. And not only the weight
bootstrap, CCR and BCC scores. In the last line, the number of efficient indices are obtained directly from the data in the data coverage analysis,
units under all three modes is shown. According to the obtained results, but this method can differentiate between the figures of the criteria
the number of efficient units of the proposed model is always lower than based on the efficiency score, as well as the inefficient resources and
the basic CCR and BCC models. values of the units under evaluation. Identify the ineffective. The main
disadvantage of data envelopment analysis models is that it has the
5. Managerial insight property of overestimating the efficiency frontier. This research aims to
present a network data envelopment analysis model to evaluate the
Data envelopment analysis is an analytical method in which avail­ performance of a sustainable supply chain using bootstrap simulation. In
able data is used to investigate causal relationships between variables. this research, using the two-step approach of data envelopment analysis
According to the results obtained in this research, the following practical and the bootstrap method, the information collected from 25 tomato
results can be presented for the research on the topic of a network data paste companies for the year 2021 has been analyzed. To illustrate the
envelopment analysis model to evaluate the performance of the sus­ proposed method, a real case study is considered in the tomato paste
tainable supply chain using bootstrap simulation: supply chain network. The findings showed that using definitive data, 16
companies are efficient and 9 companies are inefficient, and using
1. Bootstrap simulation can be used as a useful method to evaluate the bootstrap simulation data, 4 companies are efficient and 21 companies
reliability and sustainability of a sustainable supply chain. are inefficient. Therefore, the accuracy of the used method can be
2. Using the envelope analysis method and bootstrap simulation, it can concluded. The main results obtain proposed model are as follows:
be shown that the causes of inefficiency in the production process
that may have a direct impact on the stability of the supply chain can • Using the proposed framework, the overall efficiency value has been
be identified. calculated in two cases using DEA and the bootstrap model.
3. The results of this research can be used as a useful guide for supply • Also, the efficiency of each stage is calculated separately.
chain managers in making decisions related to improving perfor­ • Based on the calculated results, if a DMU is considered efficient, its
mance and promoting supply chain sustainability. efficiency score is equal to 1 in each of the stages. Otherwise, the
4. Bootstrap simulation can be used as a suitable method to evaluate the cause of the inefficiency of each DMU is identified.
reliability and correctness of the coverage analysis model.
5. Based on the research results, it can be concluded that the use of a The main limitation of the current research is the over-estimation
sustainable supply chain can improve and increase the profitability feature of existing data envelopment analysis models such as CCR and
of companies. Because it enables companies to identify the causes of BCC, which is addressed by the proposed framework based on bootstrap
inefficiency and to improve based on their weaknesses. simulation. Suggestions for future studies are suggested by the authors
6. Finally, this research can be used as a basis for further research in the to the analysts. For example, in contrast to this research, which exam­
field of sustainable supply chains and improving its performance in ined efficiency scores in convex conditions, obtain the value of efficiency
the industry. scores in non-convex conditions such as Free Disposal Hull (FDH)
models and compare the results with each other. Also, using of man­
Therefore, the main managerial insight of the current study is as agement principles along with the principle of natural accessibility can
follows: be considered attractive topics in this field.

10
M. Vaseei et al. Journal of Engineering Research xxx (xxxx) xxx

Funding [24] M. Rezaei Kallaj, M. Abolghasemian, S. Moradi Pirbalouti, M. Sabk Ara,

A. Pourghader Chobar, Vehicle routing problem in relief supply under a crisis
condition considering blood types, Math. Probl. Eng. 2021 (2021) 1–10.
This research received no specific grant from any funding agency in [25] M. Abolghasemian, A. Pourghader Chobar, M. AliBakhshi, A. Fakhr, S. Moradi
the public, commercial, or not-for-profit sectors. Pirbalouti, Delay scheduling based on discrete-event simulation for construction
projects, Iran. J. Oper. Res. 12 (1) (2021) 49–63.
[26] M. Abolghasemian, A. Ghane Kanafi, M. Daneshmandmehr, A two-phase
Declaration of Competing Interest simulation-based optimization of hauling system in open-pit mine, Iran. J. Manag.
Stud. 13 (4) (2020) 705–732.
The authors declare that they have no known competing financial [27] M. Abolghasemian, H. Darabi, Simulation based optimization of haulage system of
an open-pit mine: Meta modeling approach, Organ. Resour. Manag. Res. 8 (2)
interests or personal relationships that could have appeared to influence (2018) 1–17.
the work reported in this paper. [28] O. Kabadurmus, M.S. Erdogan, Sustainable, multimodal, and reliable supply chain
design, Ann. Oper. Res. 292 (2020) 47–70.
[29] H. Fukuyama, W.L. Weber, A slacks-based inefficiency measure for a two-stage
Availability of data and materials system with bad outputs, omega 38 (5) (2010) 398–409.
[30] S.M. Mirhedayatian, M. Azadi, S.R. Farizpoor, A novel network data envelopment
The data that support the findings of this study are available from the analysis model for evaluating green supply chain management, Int. J. Prod. Econ.
147 (2014) 544–554.
corresponding author, upon reasonable request. [31] G. Ding, Developing a multi-criteria approach for the measurement of sustainable
performance, Build. Res. Inf. 33 (1) (2007) 3–16.
References [32] Y.C. Chiou, Y.H. Chen, Route-based performance evaluation of Taiwanese domestic
airlines using data envelopment analysis, Transportation, Research 42 (2) (2006)
116–127.
[1] G.R. Jahanshahloo, A.R. Amirteimoori, S. Kordrostami, Multi-component
[33] H. Fukuyama, S.M. Mirdehghan, Identifying the efficiency status in network DEA,
performance, progress, and regress measurement and shared inputs and outputs in
Eur. J. Oper. Res. 220 (1) (2012) 85–92.
DEA for panel data: an application in commercial bank branches, Appl. Math.
[34] A. Tajbakhsh, E. Hassini, A data envelopment analysis approach to evaluate
Comput. 151 (1) (2004) 1–16.
sustainability in supply chain networks, J. Clean. Prod. 54 (7) (2014) 1–27.
[2] A. Emrouznejad, G.L. Yang, A survey and analysis of the first 40 years of scholarly
[35] M. Khodakarami, A. Shabani, R. Farzipoor sean, M. Azadi, Developing Distinctive
literature in DEA: 1978–2016, Socio-Econ. Plan. Sci. 61 (2018) 4–8.
Two stage Data envelopment analysis Models: An application in evaluating the
[3] A. Charnes, W.W. Cooper, E. Rhodes, Measuring the efficiency of decision-making
sustainability of supply chain management, Measurement 70 (2015) 62–74.
units, Eur. J. Oper. Res. 2 (6) (1978) 429–444.
[36] C. Chen, J. Zhu, J. Yu, H. Noori, A new methodology for evaluating sustainable
[4] Q.Y. Zhu, M.L. Song, J. Wu, Extended secondary goal approach for common
product design performance with two-stage network data envelopment analysis,
equilibrium efficient frontier selection in DEA with fixed-sum outputs, Comput.
Eur. J. Oper. Res. 221 (2012) 348–359.
Ind. Eng. 144 (2020), 106483.
[37] K. Simonov, H. Gupta, J. Sarkis, A Supply chain sustainability innovation
[5] W.D. Cook, R.H. Green, Evaluating power plant efficiency: a hierarchical model,
framework and evaluating methodology, Int. J. Prod. Res. 57 (7) (2019) 1–19.
Comput. Oper. Res. 32 (4) (2005) 813–823.
[38] E. Krmac, B. Djordjević, A new DEA model for evaluation of supply chains: a case
[6] W.W. Cooper, L.M. Seiford, K. Tone, Introduction to data envelopment analysis and
of selection and evaluation of environmental efficiency of suppliers, Symmetry 11
its uses: with DEA-solver software and references, Springer,, 2005.
(4) (2019) 565.
[7] C. Kao, Efficiency measurement for parallel production systems, Eur. J. Oper. Res.
[39] N. Pachar, J.D. Darbari, K. Govindan, P.C. Jha, Sustainable performance
196 (3) (2009) 1107–1112.
measurement of Indian retail chain using two-stage network DEA, Ann. Oper. Res.
[8] Dania, W.A.P., Sitepu, I.B.B., & Rucitra, A.L. (2021, April). Collaboration quality
(2021) 1–39.
assessment in the sustainable rice supply chain by using an integrated model of
[40] F. Shirazi, E. Mohammadi, Evaluating efficiency of airlines: A new robust DEA
QFD-FANP-DEA: A case study of the rice industry in Malang. In IOP Conference
approach with undesirable output, Res. Transp. Bus. Manag. 33 (2019), 100467.
Series: Earth and Environmental Science (Vol. 733, No. 1, p. 012041). IOP
[41] A. Vafaei, S. Yaghoubi, J. Tajik, F. Barzinpour, Designing a sustainable multi-
Publishing.
channel supply chain distribution network: A case study, J. Clean. Prod. 251
[9] D. Kannan, Sustainable procurement drivers for extended multi-tier context: A
(2020), 119628.
multi-theoretical perspective in the Danish supply chain, Transp. Res. Part E:
[42] M. Tavassoli, A. Fathi, R.F. Saen, Assessing the sustainable supply chains of tomato
Logist. Transp. Rev. 146 (2021), 102092.
paste by fuzzy double frontier network DEA model, Ann. Oper. Res. (2021) 1–33.
[10] A. Daneshvar, R. Radfar, P. Ghasemi, M. Bayanati, A. Pourghader Chobar, Design
[43] P.K. Dey, G.L. Yang, C. Malesios, D. De, K. Evangelinos, Performance management
of an optimal robust possibilistic model in the distribution chain network of
of supply chain sustainability in small and medium-sized enterprises using a
agricultural products with high perishability under uncertainty, Sustainability 15
combined structural equation modelling and data envelopment analysis, Comput.
(15) (2023) 11669..
Econ. 58 (3) (2021) 573–613.
[11] A. Babaeinesami, P. Ghasemi, A.P. Chobar, M.R. Sasouli, M. Lajevardi, A New
[44] T. Zhao, J. Xie, Y. Chen, L. Liang, Coordination efficiency in two-stage network
wooden supply chain model for inventory management considering environmental
DEA: application to a supplier–manufacturer sustainable supply chain, Int. J.
pollution: A genetic algorithm, Found. Comput. Decis. Sci. 47 (4) (2022) 383–408.
Logist. Res. Appl. (2021) 1–22.
[12] J. Gerami, R. Kiani Mavi, R. Farzipoor Saen, N. Kianli Mavi, A Novel Network DEA-
[45] Z. Moghaddas, B.M. Tosarkani, S. Yousefi, A developed data envelopment analysis
R Model for Evaluating Hospital Services Supply Chain Performance, Annals of
model for efficient sustainable supply chain network design, Sustainability 14 (1)
Operations Research,, 2020.
(2021) 262.
[13] M. Ansari, J.S. Borrero, L. Lozano, Robust minimum-cost flow problems under
[46] Q. Chen, S. Chen, D. Liu, Regret-based cross efficiency evaluation method in a
multiple ripple effect disruptions, INFORMS J. Comput. 35 (1) (2023) 83–103.
general two-stage DEA system, Comput. Ind. Eng. 175 (2023), 108828.
[14] F. Asadia, M. Abolghasemianb, Review coordination of advertising policy and its
[47] H. Fitzová, M. Matulová, Comparison of urban public transport systems in the
effect on competition between retailer and manufacture in the supply chain,
Czech Republic and Slovakia: Factors underpinning efficiency, Res. Transp. Econ.
Comput. Res. Prog. Appl. Sci. Eng. 4 (3) (2018) 62–66.
81 (2020), 100824.
[15] S.P. Singh, M. Song, R.R.K. Sharma, R. Fisher, R.K. Amit, R. Farzipoor Saen,
[48] L. Yan, C. Ma, Y. Yang, E. Zhan, H. Lv, Estimating the regional eco-efficiency in
Preface: Sustainable operations in a manufacturing enterprise, Ann. Oper. Res. 290
china based on bootstrapping by-production technologies, J. Clean. Prod. 243
(2020) 1–4.
(2020) 1–13.
[16] K. Simonov, H. Gupta, J. Sarkis, A Supply chain sustainability innovation
[49] M.N. Nodin, Z. Mustafa, S.I. Hussain, Assessing rice production efficiency for food
framework and evaluating methodology, Int. J. Prod. Res. 57 (7) (2019) 1–19.
security policy planning in Malaysia: A non-parametric bootstrap data
[17] H. Golmoradi, S. Golzarian Pour, S. Aliakbar, The Effect of Bank Survival Factor on
envelopment analysis approach, Food Policy 107 (2022), 102208.
Banking Cost Efficiency in Iranian Banks, Q. J. Quant. Econ. (JQE) 17 (4) (2021)
[50] Y. Yu, M. Zhang, B. Huo, The impact of relational capital on green supply chain
89–111.
management and financial performance, Prod. Plan. Control 32 (10) (2021)
[18] M.J. Farrell, The measurement of productive efficiency, J. R. Stat. Soc. 120 (1957)
861–874.
253–281.
[51] A. Singh, H. Gundimeda, Analysing drivers of efficiency in the leather industry: a
[19] J. Zhu, DEA Under big Data: Data Enabled Analytics and Network Data
two-stage double bootstrap DEA approach, Benchmark.: Int. J. 29 (9) (2022)
Envelopment Analysis, Annals of Operations Research,, 2020.
2780–2805.
[20] J.S. Sun, S.Q. Xu, G. Li, Analyzing sustainable power supply chain performance:
[52] S.H. Hashemi, A. Karimi, M. Tavana, An integrated green supplier selection
evidence from China’s provincial regions, J. Enterp. Inf. Manag. 34 (1) (2020)
approach with analytic network process and improved Grey relational analysis, Int.
79–100.
J. Prod. Econ. 159 (2015) 178–191.
[21] Vaez-Ghasemi, Moghaddas, Z., Farzipoor Saen, R., 2021, Cost efficiency evaluation
[53] F. Asadi, S. Kordrostami, A. Amirteimoori, M. Bazrafshan, Inverse data
in sustainable supply chains with marginal surcharge values for harmful
envelopment analysis without convexity: double frontiers, Decis. Econ. Financ. 46
environmental factors: a case study in a food industry. Operational Research.
(1) (2023) 335–354.
[22] R. Rajesh, Sustainable supply chains in the Indian context: an integrative decision-
making model, Technol. Soc. 61 (2020), 101230.
[23] S. Jahangiri, M. Abolghasemian, P. Ghasemi, A.P. Chobar, Simulation-based
optimisation: analysis of the emergency department resources under COVID-19
conditions, Int. J. Ind. Syst. Eng. 43 (1) (2023) 1–19.

11
M. Vaseei et al. Journal of Engineering Research xxx (xxxx) xxx

[54] F. Asadi, S. Kordrostami, A. Amirteimoori, M. Bazrafshan, Inverse data [56] Heydari Kushalshah, T., Daneshmand-Mehr, M., & Abolghasemian, M. (2023).
envelopment analysis on the base of non-convex cost efficiency, Iran. J. Oper. Res. Hybrid modelling for urban water supply system management based on a bi-
13 (1) (2022) 135–148. objective mathematical model and system dynamics: A case study in Guilan
[55] M. Abolghasemian, A.G. Kanafi, M. Daneshmand-Mehr, Simulation-based province. Journal of Industrial and Systems Engineering.
multiobjective optimization of open-pit mine haulage system: a modified-NBI
method and meta modeling approach, Complexity (2022) 2022.

12