Portland State University

PDXScholar

Engineering and Technology Management Engineering and Technology Management

Faculty Publications and Presentations

1-1-1997

An Introduction To Data Envelopment Analysis In

Technology Management
Keith Hollingsworth
Morehouse College

Timothy R. Anderson
Portland State University, tim.anderson@pdx.edu

Follow this and additional works at: https://pdxscholar.library.pdx.edu/etm_fac

Part of the Engineering Commons

Let us know how access to this document benefits you.

Citation Details
Anderson, T.R., "An Introduction To Data Envelopment Analysis In Technology Management," Innovation in
Technology Management - The Key to Global Leadership. PICMET '97: Portland International Conference
on Management and Technology, pp.773.

This Article is brought to you for free and open access. It has been accepted for inclusion in Engineering and
Technology Management Faculty Publications and Presentations by an authorized administrator of PDXScholar.
Please contact us if we can make this document more accessible: pdxscholar@pdx.edu.
 1997 Proceedings of PICMET '97: Innovation in Technology Management: The Key to Global Leadership

An Introduction to Data Envelopment Analysis in Technology Management

Timothy R. Anderson Keith B. Hollingsworth

Engineering Management Program Department of Economics and Business Administration
Portland State University Morehouse College
Portland, OR 97207-0751 USA Atlanta, GA 30314 USA

Abstract-Productivity is a major source of competitive some of the most important modeling variations, and examples
advantage but improving productivity requires analysis and of applications relevant to technology management.
productivity is getting more difficult to measure as economies We will focus upon the so-called radial models. While
become more knowledge, service, and innovation intensive. In there are additional models such as the additive model and
this paper, we will provide an introduction to a powerful multiplicative models, the radial models have been the focus
productivity analysis tool, Data Envelopment Analysis (DEA).
of the vast majority of the applications as well as the important
DEA is a flexible tool originally created in the 1970s for
examining the relative efficiency of non-profit institutions. modeling extensions.
Researchers have found many more applications and created
numerous extensions to DEA. This paper provides an II. DEVELOPMENT OF THE BASIC DEA MODELS
introduction to DEA, a summary of some of the most
important modeling variations, and examples of applications A. Background
relevant to technology management.
Although the origin of DEA is generally accepted as
beginning with the seminal work of [16], like most important
I. INTRODUCTION scientific discoveries, it builds on a strong foundation of
developments in the area of production theory such as the
A. Why Study Productivity Analysis works of Farrell [18, 20], Koopmans [29], and others.
As with most optimization based systems, there are two
Peter Drucker has recently stated [19] that “…productivity fundamentally different approaches that yield essentially
is the true source of competitive advantage.” While it is a equivalent solutions. These approaches correspond to the
common syndrome that many people become enamored with primal and dual formulations. We will trace our way through
technology for the sake of technology itself, the real purpose of an intuitive explanation of both formulations. An intuitive
technology is to directly or indirectly improve productivity. understanding of the interpretations for both formulations is
Productivity is a major source of competitive advantage but important since they each have subtle implications for how
improving productivity requires analysis and productivity is well a DEA analysis actually fits a particular application. The
getting more difficult to measure as economies become more selection of which model is the primal and which is the dual is
knowledge, service, and innovation intensive. Traditional of course, arbitrary, so we will follow the conventions set
productivity measures require aggregating everything into a forth in [15] which has attempted to standardize the notation of
single measure such as money. This aggregation presupposes DEA.
known prices for factors of production. While the challenge of
determining prices was tractable in the past when labor and B. Numerical Example
capital were the primary constituents of production, it is
becoming increasingly difficult. A single set of costs is being A numerical example will be used to help illustrate the
made more difficult by the increasingly international various DEA models. Consider the case of evaluating a set of
orientation of most companies, knowledge intensive products, software development teams. Each team is made up of a
shifts from product to service orientations, complex number of junior and senior programmers which will be
organizational structures including outsourcing and virtual termed the inputs. The accomplishments (or outputs) of the
organizations, and the shortening of product life cycles. teams are listed by total number of lines of code and by the
In this paper, we will provide an introduction to a powerful number of projects undertaken. Further refinements to these
productivity analysis tool, Data Envelopment Analysis (DEA). measures could be easily made by including the years of
DEA is a flexible tool originally created in the 1970s for experience of the programmers or categorizing the difficulty of
examining the relative efficiency of non-profit institutions. projects. This additional complexity would not prevent the use
Since that time, researchers have found many more of DEA but would make an intuitive explanation more difficult
applications and created numerous extensions to DEA. A so this simple model will suffice for the sake of illustration.
major advantage of DEA is that it is not based on knowing the We should therefore assume for now that all of the junior
production function thus allowing for richer models and does programmers are essentially equally skilled; all of the senior
not rely upon knowledge of the prices of production factors. programmers are essentially equally skilled; and a line of code
This paper provides an introduction to DEA, a summary of has a consistent complexity. This data is summarized in the

Refereed Paper
 1997 Proceedings of PICMET '97: Innovation in Technology Management: The Key to Global Leadership

following table. The selection of inputs and outputs should by which the input of the virtual team can be reduced and still
never be viewed as a trivial task. On the contrary, it is a subtle produce at least as much of each output at the team being
issue which requires a thorough knowledge of the application studied. Therefore, θ indicates the input-oriented radial
area. As a basic rule, inputs are “bads” and outputs are relative efficiency of the team.
“goods.” For further discussion, see [33, 43].
TABLE I min θ ,
SOFTWARE PROGRAMMING TEAMS θ, λ
s.t. Y λ ≥ Y0 , (1)
Junior Senior Lines of
Team
Programmers Programmers Code
Projects Xλ ≤ θ X 0 ,
1 6 3 12500 4 λ , ≥ 0.
2 5 2 13000 5
3 7 4 14500 8
An efficiency score of θ = 1 for a particular team indicates
4 3.5 2 8000 4
6 3 13250 6
that there is no possible DMU (real or virtual) that can
5
produce at least as much of each output using no more of any
input than this team. We would then deem this team to be
Since we assume each team has the freedom to decide how
radially efficient. Our results are in Table II.
to best allocate their inputs to produce outputs, we will refer
to them as decision making units (DMUs.) Our goal is to TABLE II
measure the productivity of each software development team. SIMPLE RADIAL CCR INPUT-O RIENTED PRIMAL RESULTS
This would enable us to determine things such as what are the Team λ1 λ2 λ3 λ4 λ5 Efficiency
“best practices” in the area of software development, which 1 0.962 0.1 0.801
teams should be rewarded, what are the performance targets 2 1 1.000
they are given, etc. 3 1 1.000
4 1 1.000
C. CCR Primal Formulation 5 0.462 0.923 0.923

Perhaps the first approach to come to mind would be to see Not all teams that are radially efficient are really making the
if any team used more inputs (programmers) to produce less best use of their inputs to make outputs though. For example
outputs (both lines of code and number of projects) than one of compare Team 3, which was radially efficient, to a doubled
the other teams. In our example, this would be the case of Team 4. Both Team 3 and the doubled Team 4 had the same
Team 1 being outperformed (or dominated) by Team 2. Note number of programmers and completed the same number of
that all of the other teams are undominated. This technique projects but Team 3 finished less lines of code. This indicates
does not provide a lot of discriminating power. While this is slack in that output for Team 3. We should then expand our
an important case, it is relatively rare that DMUs are purely definition of efficiency to include only those teams (or DMUs)
dominated by a single other DMU. Still, this technique has that are radially efficient and do not use any slack inputs or
found useful application as demonstrated by [47]. outputs. After determining the maximum radial contraction
Rather than limit ourselves to only those teams that actually possible (minimum θ), we should try to maximize the slacks of
exist, if we assume that “composite” or “virtual” teams could the inputs and the outputs.
be formed from multiples of the other teams, it considerably Notationally, we can combine these two steps into a single
broadens our possible comparison set. Each team (or DMU) is
linear programming formulation using an infinitesimal, ε, that
then compared against a best possible virtual team for that
is greater than zero but smaller than any real positive number.
particular team. These comparison teams, or peers, form a
This enables the minimization of the radial efficiency score to
frontier against which a particular team is measured. Let us
take precedence over the maximization of the slacks.
assume that the goal is to minimize the inputs used to produce a + -
certain level of output. This is a plausible goal since we could min+ -
θ - εs - εs ,
θ , λ, s , s
find other tasks for skilled programmers.
+
Determining this best possible virtual team for each team is s.t. Y λ − Y0 - s = 0, (2)
not difficult in two dimensions but it becomes very difficult in -
higher order situations. This problem is much better analyzed θ X 0 − X λ − s = 0,
+ -
as a linear optimization problem. The matrices X and Y refer to λ , s , s ≥ 0.
the sets of inputs and outputs respectively. The vector λ
describes the construction of the virtual team used in Attempting to approximate ε by a small number, however,
comparison with the particular team under consideration. The may lead to subtle computational problems as documented in
vectors X0 and Y0 are the inputs and outputs of the particular [1]. Therefore, it is often solved as a two-phase procedure.
team (or DMU) being analyzed. The value, θ, is the proportion The two-phase procedure is really just a simple example of

Refereed Paper
 1997 Proceedings of PICMET '97: Innovation in Technology Management: The Key to Global Leadership

preemptive goal programming. The objective of the first phase themselves in the best possible light. The one restriction is that
(linear program) is to minimize θ. The second phase’s no team can pick a pricing (or weighting) scheme that would
objective is to minimize the negative sum of slacks (or give any other team a score greater than one. Clearly a team
maximize the sum of slacks) given that θ must be unchanged that does a lot of short projects might then want to put a higher
from the result found in the first phase. Although Team 3 is weight on the number of projects than on the lines of code.
radially efficient, it was only weakly efficient and is Similarly, a team of junior programmers might put a far higher
outperformed by Team 4. This would result in a higher target weight on senior programmers than on junior programmers to
of performance for Team 3 based upon Team 4’s performance. make themselves look better.

III. ADDITIONAL STANDARD MODELS µ ⋅ Y0

max ,
v, µ v⋅ X0
A. CCR Dual Formulation
µ ⋅ Yj
s.t. ≤ 1, for each DMU j∈[1,..., n] (3)
Traditionally, each software development team might be v⋅ X j
rated by some ratio of aggregated outputs over aggregated
inputs. As discussed earlier, financial values often provide a µ, v ≥ 0.
useful method for aggregation. The programmers’ salaries
could be used in this case to aggregate the inputs. Assume that Although this problem is a nonlinear function, as originally
the average annual salary for a senior programmer is $70,000 demonstrated in [13] it can be easily linearized by multiplying
and for junior programmer, $40,000. The outputs might be out the denominator of each constraint and by setting the
harder to aggregate since it may be hard to relate lines of code denominator of the objective function to be equal to one.
to a dollar figure. Let us assume each project brings in revenue Lastly, we would like to eliminate the possibility of anyone
of $100,000 and a line of code is valued at an additional $100. achieving a determination of being efficient by disregarding
We could now easily calculate aggregate inputs and outputs. one of the inputs or outputs entirely. This situation can be
From these aggregate values, we could then calculate a yield resolved by including a constraint that each input and output
and relative rating of each team on the basis of the yield. This weight must be greater than or equal to an infinitesimal. The
type of procedure would be a fixed-weight measure. resulting linear program is given in (4).
TABLE III
FIXED WEIGHT EVALUATION max µ ⋅ Y0 ,
v, µ

Team
Salary Deliverable
Yield Rating s.t. v ⋅ X 0 = 1,
(4)
[ ]
Cost ($K) Value ($K)
1 840 1650 1.96 0.69 µ ⋅ Yj ≤ v ⋅ X j , for each DMU j ∈ 1,..., n
2 630 1800 2.86 1.00
µ , v ≥ ε.
3 1050 2250 2.14 0.75
4 525 1200 2.29 0.80
5 840 1925 2.29 0.80 B. BCC Primal Formulation

When looking at the results in Table II, we see the effects of

If we have accurate prices or weights for each input and
another major assumption of the DEA model as written. Team
output, this can be a very easy and valuable technique. (See
3 is compared to the doubled Team 4. This assumes that Team
[26] for a nice and succinct overview of some fixed-weight
4 will operate just as efficiently at twice its current size. In
techniques productivity analysis techniques.) In this example,
essence, the model assumes that doubling the programmers on
it appears to be straightforward to aggregate both the inputs
a project will double the code produced. This assumption is
and the outputs through a common set of units such as dollars.
termed constant returns to scale. However, this is a major and
The example could be easily modified to make it much more
often unrealistic assumption. In the application of software
difficult to aggregate. For example, quality measures could be
development, it has been stated [11] that adding programmers
included as an output, development tools could be included as
seldom yields an equivalent increase in output thereby
an input, and salaries could include international programmers
displaying decreasing returns to scale. For more detailed
with different salary units of pay. In these cases, the tradeoffs
discussions of the economies of scale for software
between inputs or the tradeoffs between outputs are much more
programming, see [4, 5, 7].
difficult to make.
The assumption of constant returns to scale in the primal
The dual formulation of DEA can be thought of as being
analogous to the increasingly popular method of annual review formulation is a result of the freedom of the λ’s in constructing
known as “self-evaluation.” We will allow each team the targets of performance. Banker, Charnes, and Cooper, [6]
opportunity to rate themselves by selecting the “prices” for demonstrated a simple way to unrestrict the returns to scale
each input, v, and the “prices” for each output, µ, that cast assumption was to restrict the values of the λ’s. As in the case
of the CCR model, this model is widely referred to by the

Refereed Paper
 1997 Proceedings of PICMET '97: Innovation in Technology Management: The Key to Global Leadership

author’s initials of BCC. In essence, it is based on restricting mentioned in section D to place all the weight on the senior
how virtual teams are created. By adding the restriction that programmers because we know from experience that junior
the λ’s must sum to one, all virtual teams must lie on a plane programmers have the high cost of training and turnover. So
between actual teams. This restriction of convexity creates a we may restrict the junior programming weight to be over a
much tighter frontier surrounding the data and a more varied minimum proportion of the senior programmers’ cost.
surface. This surface can now display different returns to scale So, the restriction of these weights allows the expert
at different locations. The new formulation is shown below: opinion of the analyst to be included in the model. It is
important to recognize that not using a model that allows for
+ - weight restrictions implicitly assumes that there is no limit to
min+ θ - εs - εs ,
θ , λ, s , s
- the relative weights of the inputs or of the outputs.
+ Weight restriction techniques have another major advantage.
s.t. Y λ − Y0 - s = 0, Limiting the freedom of a unit to select unrealistic weights
-
θ X 0 − Xλ − s = 0, (5) assists the analyst in performing realistic evaluations with a
n
limited number of DMUs. A general rule of thumb is one
∑ λ j = 1, should have a set of DMUs equal to three times the sum of the
j=1 number of inputs and outputs in a model. For a further
+ - discussion of this which considers the motives behind the
λ , s , s ≥ 0. model see [2]. This paper, however, does not consider the
impact of weight restrictions.
By looking at Table IV, we can see how this affects our A method of restricting weights in DEA that does not
problem. Team 3 is still considered efficient but now lies on require a priori knowledge of weights for a particular
the frontier rather than being compared to a doubled Team 4. application is cross-efficiency. The use and implications of
Teams 1 and 5 are still inefficient but have slightly higher cross-efficiency are subtle and are discussed in more detail in
efficiency scores than with the previous analysis. This occurs other PICMET papers.
because the frontiers to which they are compared have been
restricted into tighter areas. However, the main change lies in C. Types of Inputs and Outputs
the teams used as comparisons.
DEA assumes that the inputs and outputs have a variety of
TABLE IV properties including being quantifiable and controlled by the
BCC INPUT-O RIENTED PRIMAL EVALUATION DMU. It has been found that in certain applications one or
more of the inputs or outputs does not satisfy these properties.
Team λ1 λ2 λ3 λ4 λ5 Efficiency Difficult to quantify inputs and outputs have been typically
1 0.9 0.1 0.808 addressed by the use of models that allow for categorical
2 1 1.000 variables. Categorical variables are those having a type such
3 1 1.000
as “small, medium or large” rather than a quantifiable
4 1 1.000
valuable. The issue of categorical inputs and outputs has been
5 0.593 0.352 0.056 0.937
examined by a number of researchers [9, 17, 23, 27, 38]. The
basic technique is to restrict the comparison set. An example
IV. IMPORTANT MODELING VARIATIONS AND EXTENSIONS using an ordinal categorical variable could be found in the
application of software programmers. Consider using the
A. Overview
categorical input variable of location with two values: “on-
Since the development of DEA, several variations have site” and “off-site”. Since the on-site programmers
been proposed which assist in refining the accuracy of the presumably have access to far better equipment and more
model. These variations and extensions take several forms. We resources, we might not want to include them in the
will address methods of weight restrictions and nontraditional comparison group for the “off-site” teams. Therefore, all “off-
types of inputs and outputs. Work is being conducted to site” teams would only be compared to each other. However,
develop and improve statistical interpretations to DEA results. the “on-site” teams would be compared to the entire set of
programming teams.
B. Weight Restrictions To illustrate the issue of discretionary versus
nondiscretionary inputs and outputs, we consider again our
An important extension allows the analyst to limit the range software programmers. Suppose this model was used to
in which the weights or “prices” can vary in the dual measure the effectiveness of the senior programmers at
formulation. The dual formulation allows each DMU to choose managing the programming process. If so, they might not have
the set of weights which gave it the highest score. However, any control over the number of senior programmers on their
the analyst may possess knowledge to the contrary. For team; this having been decided at higher levels. The senior
example, we may not allow the junior programming team programmers would only be able to control the number of

Refereed Paper
 1997 Proceedings of PICMET '97: Innovation in Technology Management: The Key to Global Leadership

junior programmers assigned to the team. Therefore, a model warehousing operations [21, 25, 39]. As has already been
judging their performance should distinguish between the discussed, software development has been examined [4, 5, 7]
inputs under their control (the discretionary inputs) and the and continues to receive attention as this sector of the economy
inputs beyond their jurisdiction (the nondiscretionary inputs). has grown in importance and complexity. The changing
A variety of authors [8, 14, 22, 37] have motivated and paradigms of manufacturing have also resulted in a need for
developed models that account for nondiscretionary or powerful analytical tools including DEA [12, 24, 36, 41, 42,
partially discretionary inputs and/or outputs. 49]. Recently, DEA has been brought to bear on the subject of
technology selection and evaluation [3, 28]. As previously
V. CALCULATING DEA moribund industries deregulate, efficiency analyses will assist
them to determine ways to operate better [10, 23, 30, 31, 40,
A. Computational Difficulties 44, 45, 48, 50]. Product life cycle and marketing has also seen
recent contributions made using DEA [32, 46] as well as R&D
The linear programming formulations for DEA may at first project selections [35].
glance appear to be rather simple. In fact it is simple enough
that it is now often included in introductory management VII. CONCLUSIONS
science textbooks. This may lead people to rely on develop
their own software for performing DEA. Unfortunately, the Data envelopment analysis has been a useful tool that is
simplicity of the formulations belie more subtle computational starting to see application in areas of the management of
issues. DEA problems actually have a number of relatively engineering and technology. It will continue to grow in
uncommon characteristics that may cause difficulty. These importance as analysts apply DEA in a variety of settings that
problems tend to have highly degenerate solutions. For require refinements.
example, in the primal formulation it is not unusual to have a
solution where there are far fewer nonzero λ values than there REFERENCES
are input and output constraints. Also, although DEA problems
tend to be much smaller than large scale optimization [1] A. I. Ali, “Data envelopment analysis: computational issues,” Computers,
problems, data matrices are nearly 100% dense as opposed to Environment and Urban Systems, vol. 14, pp. 157-65, 1990.
[2] T. R. Anderson, K. B. Hollingsworth, and M. L. Durchholz, “Estimates of the
the sparse matrices that commercial linear programming
Number of DMUs Needed for DEA as a Function of Model Size,” Working
software is traditionally tailored to address. Lastly, as Paper, pp. 1-17, 1996.
discussed earlier, it is important to incorporate a two-phase [3] R. C. Baker and S. Talluri, “A closer look at the use of data envelopment
approach rather than attempting to approximate a non- analysis for technology selection,” Computers & Industrial Engineering, vol.
Archimedean infinitesimal. 32, pp. 101-108, 1997.
[4] R. Banker and C. Kemerer, “Factors affecting software maintenance
B. Software Systems productivity: an exploratory study,” presented at Proceedings of the 8th
International Conference on Information Systems, Pittsburgh, PA, 1987.
[5] R. D. Banker, H. Chang, and C. F. Kemerer, “Evidence on economies of scale
Since the use of DEA has not been as widespread in the
in software development,” Information & Software Technology, vol. 36, pp.
statistics community, there has not been a great demand to 275-282, 1994.
include it in standard research support packages. Instead, a [6] R. D. Banker, A. Charnes, and W. W. Cooper, “Some models for estimating
variety of systems exist which specifically perform DEA. A technical and scale inefficiencies in data envelopment analysis,” Management
couple of systems with graphical user interfaces such as Science, vol. 30, pp. 1078-1092, 1984.
Warwick-DEA and Frontier Analyst have recently been [7] R. D. Banker and C. F. Kemerer, “Scale economies in new software
developed that incorporate a variety of common models and development,” IEEE Transactions on Software Engineering, vol. 15, pp.
1199-205, 1989.
extensions. Another option that allows the user to more easily [8] R. D. Banker and R. C. Morey, “Efficiency analysis for exogeneously fixed
extend models is to use a template in a general purpose inputs and outputs,” Operations Research, vol. 34, pp. 513-21, 1986.
algebraic modeling languages. Basic templates have been [9] R. D. Banker and R. C. Morey, “The use of categorical variables in data
developed in GAMS [34] and AMPL. Lastly, special purpose, envelopment analysis,” Management Science, vol. 32, pp. 1613-27, 1986.
batch oriented programs have also been developed that take [10] W. F. Bowlin, “A characterization of the financial condition of the United
special advantage of structures in the DEA problems. States' aerospace-defense industrial base,” Omega, vol. 23, pp. 539-55,
1995.
[11] F. P. J. Brooks, The Mythical Man-Month - Essays on Software
VI. TECHNOLOGY MANAGEMENT APPLICATIONS Engineering. Reading: Addison-Wesley, 1982.
[12] Y.-L. Chang, T. Sueyoshi, and R. S. Sullivan, “Ranking dispatching rules by
Initially DEA was developed for use in the evaluation of data envelopment analysis in a job shop environment,” IIE Transactions, vol.
nonprofit institutions where it was difficult or meaningless to 28, pp. 631-642, 1996.
attempt to aggregate inputs and outputs into a simple [13] A. Charnes and W. W. Cooper, “Programming with linear fractional
bottomline profit/loss situation. Analysts, however, have found functionals,” Naval Research Logistics Quarterly, vol. 9, pp. 181-185, 1962.
it to be a useful technique in a wide range of areas. DEA was [14] A. Charnes, W. W. Cooper, B. Golany, and L. Seiford, “Foundations of
data envelopment analysis for Pareto-Koopmans efficient empirical production
used to evaluate the performance of company subunits, such as
functions,” Journal of Econometrics, vol. 30, pp. 91-107, 1985.

Refereed Paper
 1997 Proceedings of PICMET '97: Innovation in Technology Management: The Key to Global Leadership

[15] A. Charnes, W. W. Cooper, A. Y. Lewin, and L. M. Seiford, “Data [33] O. B. Olesen and N. C. Petersen, “Indicators of ill-conditioned data sets
Envelopment Analysis: Theory, Methodology and Applications,” . Boston: and model misspecification in data envelopment analysis: An extended facet
Kluwer, 1994. approach,” Management Science, vol. 42, pp. 205-219, 1996.
[16] A. Charnes, W. W. Cooper, and E. Rhodes, “Measuring the efficiency of [34] O. B. Olesen and N. C. Petersen, “A presentation of GAMS for DEA,”
decision making units,” European Journal of Operations Research, vol. 2, Computers & Operations Research, vol. 23, pp. 323-339, 1996.
pp. 429-44, 1978. [35] M. Oral, O. Kettani, and P. Lang, “A methodology for collective evaluation
[17] A. Charnes, J. J. Rousseau, and J. H. Semple, “Nonarchimedean and selection of industrial R&D projects,” Management Science, vol. 37, pp.
infinitesimals, transcendentals and categorical inputs in linear programming and 871-885, 1991.
data envelopment analysis,” International Journal of Systems Science, vol. [36] C. Parkan, “Calculation of operational performance ratings,” International
23, pp. 2401-6, 1992. Journal of Production Economics, vol. 24, pp. 165-1 73, 1991.
[18] G. Debreu, “The coefficient of resource utilization,” Econometrica, vol. 19, [37] S. C. Ray, “Data envelopment analysis, nondiscretionary inputs and
pp. 273-292, 1951. efficiency: an alternative interpretation,” Socio-Economic Planning Sciences,
[19] P. F. Drucker, “The new productivity challenge,” Harvard Business vol. 22, pp. 167-176, 1988.
Review, vol. 69, pp. 69-79, 1991. [38] J. J. Rousseau and J. H. Semple, “Notes: Categorical outputs in data
[20] M. J. Farrell, “The measurement of productive efficiency,” Journal of the envelopment analysis,” Management Science, vol. 39, pp. 384-386, 1993.
Royal Statistical Society, vol. 120, pp. 253-281, 1957. [39] M. Schefczyk, “Industrial benchmarking: a case study of performance
[21] E. H. Frazelle and S. T. Hackman, “The warehouse performance index: a analysis techniques,” International Journal of Production Economics, vol.
single-point metric for benchmarking warehouse performance,” Material 32, pp. 1, 2-11, 1993.
Handling Research Center MHRC-TR-93-14, 9/20/94 1994. [40] M. Schefczyk, “Operational performance of airlines: an extension of
[22] B. Golany and Y. Roll, “Some extensions of techniques to handle non- traditional measurement paradigms,” Strategic Management Journal, vol.
discretionary factors in data envelopment analysis,” Journal of Productivity 14, pp. 301-317, 1993.
Analysis, vol. 4, pp. 419-432, 1993. [41] S. M. Shafer and J. W. Bradford, “Efficiency measurement of alternative
[23] B. Golany, Y. Roll, and D. Ryback, “Measuring efficiency of power plants machine component grouping solutions via data envelopment analysis,” IEEE
in Israel by data envelopment analysis,” IEEE Transactions on Engineering Transactions on Engineering Management, vol. 42, pp. 159-165, 1995.
Management, vol. 41, pp. 291-301, 1994. [42] J. Shang and T. Sueyoshi, “A unified framework for the selection of a
[24] K. T. Heimerman, “Efficient Set Relations Among Data Envelopment flexible manufacturing system,” European Journal of Operational Research,
Analysis Models and Resource Use Efficiency In Manufacturing,” : University vol. 85, pp. 297-315, 1995.
Of Massachusetts, 1993, pp. 362. [43] P. Smith, “Misspecification bias in data envelopment analysis,” University of
[25] K. B. Hollingsworth, “A Warehouse Benchmarking Model Utilizing Frontier York, Discussion Papers 15, May 1993 1993.
Production Functions,” in Industrial and Systems Engineering. Atlanta: [44] T. Sueyoshi, “Divestiture of Nippon Telegraph and Telephone,”
Georgia Institute of Technology, 1995. Management Science, vol. 42, pp. 1326-1351, 1996.
[26] J. Jurison, “Defining and Measuring Productivity,” in Productivity in the [45] D. T. Taylor and R. G. Thompson, “The efficient utility: Labor, capital, &
Office and Factory, Decision Making and Operations Management Series, profit,” Public Utilities Fortnightly, vol. 133, pp. 25-29, 1995.
P. Gray and J. Jurison, Eds. Danvers: Boyd and Fraser, 1995, pp. 11-21. [46] S. Thore, F. Phillips, T. W. Ruefli, and P. Yue, “DEA and the management
[27] W. A. Kamakura, “A note on 'the use of categorical variables in data of the product cycle: The U.S. computer industry,” Computers & Operations
envelopment analysis',” Management Science, vol. 34, pp. 1273-6, 1988. Research, vol. 23, pp. 341-356, 1996.
[28] M. Khouja, “The use of data envelopment analysis for technology [47] H. Tulkens, “On FDH efficiency analysis: some methodological issues and
selection,” Computers & Industrial Engineering, vol. 28, pp. 123-32, applications to retail banking, courts, and urban transit,” Journal of
1995. Productivity Analysis, vol. 4, pp. 183-210, 1993.
[29] T. C. Koopmans, “An analysis of production as an efficient combination of [48] L. C. Walters, G. C. Cornia, D. W. Shank, C. Gerschefske, and et al.,
activites,” in Activity Analysis of Production and Allocation, vol. 13, “Measuring obsolescence in regulated firms: Enhancements to the cost
Cowles Commission for Research in Economics, T. C. Koopmans, Ed. approach,” Assessment Journal, vol. 1, pp. 47-58, 1994.
New York: John Wiley and Sons, Inc., 1951. [49] C. H. Wang, “The Impact Of Manufacturing Performance On Firm
[30] S. K. Majumdar, “X-efficiency in emerging competitive markets: The case Performance, the Determinants Of Manufacturing Performance, and the Shift
of U.S. telecommunications,” Journal of Economic Behavior & Of the Manufacturing Efficiency Frontier,” : State University Of New York At
Organization, vol. 26, pp. 129-144, 1995. Buffalo, 1993, pp. 233.
[31] S. K. Majumdar and H.-h. Chang, “Scale efficiencies in US [50] S. Yaisawarng and J. D. Klein, “The effects of sulfur dioxide controls on
telecommunications: An empirical investigation,” Managerial & Decision productivity change in the U.S. electric power industry,” Review of Economics
Economics, vol. 17, pp. 303-318, 1996. & Statistics, vol. 76, pp. 447-460, 1994.
[32] P. S. Murthi, K. Srinivasan, and G. Kalyanaram, “Controlling for observed
and unobserved managerial skills in determining first-mover market share
advantages,” Journal of Marketing Research, vol. 33, pp. 329-336, 1996.

Refereed Paper