TELEMATICS and INFORMATICS Vol. 3, No. 4, pp. 289-300, 1986 Copyright c, 1987 Pergamon Journals Ltd. Printed in the USA 0736-5853/86 $3.00 + .00 R&D PROJECT SELECTION Matthew d. Liberatore Abstract-The research and development project selection decision is concerned with the allocation of resources to a set of proposals for scientific and engineering activities. The project selection process can be viewed as a multiple-criteria decision-making problem, within the context of the long-range and strategic planning process of the firm. The purpose of this paper is explore the applicability of several approaches, including the Analytic Hierarchy Process, for priority setting and resource allocation in the industrial R&D environment. The incorporation of these models into expert support systems for R&D project selection is discussed. INTRODUCTION The complexity of the R&D project selection decision can be understood by a recognition of several key issues and factors. Project selection decisions affect the current and future financial position of the organization, and its ability to compete on a technological basis. Yet the returns from R&D activities often occur several years into the future and are risky in terms of projected outcome. These returns are multidimensional in nature and include other factors besides clearly measurable financial outcomes. For these reasons, R&D project selection decisions should be linked to the strategic objectives and plans of the firm. A variety of models and methods for R&D project selection and resource allocation have been developed.~.'-.4 A recent empirical study on the current usage of quantitative techniques for R&D project management in "Fortune 500" industrial firms was conducted? Key results concerning project selection and resource allocation include: 1) a heavy usage of financial analysis techniques; 2) minimal usage of mathematical programming models; and 3) mixed usage of budgeting systems based on cost/benefit tradeoffs. Many R&D managers do not perceive that the available methods for project selection and resource allocation appreciably improve their decision-making. These results lead to three related conclusions: 1. The organizational context in which R&D project selection and resource allocation occurs must be considered in the development of appropriate methods and systems. For example, the availability of data for measuring costs and benefits, the statement of organizational and project goals, the criteria for selection, and the structure of R&D and supporting groups, including information flows, a m o n g other factors, vary across industrial firms. 2. Project selection is invariably based on economic as well as social benefit-cost analysis. This involves the consideration of both quantitative and qualitative factors. Dr. Liberatore is a Professor in the Department o f Management, College o f Commerce and Finance, Villanova University. 289 290 Matthew J. Liberatore 3. Methods which provide for the measurement and aggregation of the various project selection criteria seem most appropriate for prioritizing and ranking projects. Based on the above considerations four classes of multi-criteria methods are reviewed: scoring models, goal programming(GP), multi-attribute utility theory(MAU), and the analytic hierarchy process(AHP). Each could form an important component of an expert support system for R&D project selection. FACTORS NO. OF COMPANIES RESEARCH AND DEVELOPMENT LIKELIHOOD OF TECHNICAL SUCCESS DEVELOPMENT COST DEVELOPMENT TIME CAPABILITY OF AVAILABLE SKILLS AVAILABILITY OF R&D RESOURCES AVAILABILITY OF R&D FACILITIES PATENT STATUS COMPATIBILITY WITH OTHER PROJECTS 15 10 8 7 5 3 3 2 MANUFACTURING CAPABILITY OF MANUFACTURING PRODUCT FACILITY AND EQUIPMENT REQUIREMENTS AVAILABILITY OF RAW MATERIAL MANUFACTURING SAFETY 12 6 2 1 MARKETING AND DISTRIBUTION SIZE OF POTENTIAL MARKET CAPABILITY TO MARKET PRODUCT MARKET TREND AND GROWTH CUSTOMER ACCEPTANCE RELATIONSHIP WITH EXISTING MARKETS MARKET SHARE MARKET RISK DURING DEVELOPMENT PERIOD PRICING TREND, PROPRIETARY PROBLEM, GEOGRAPHICAL EXTENT, AND EFFECT ON EXISTING PRODUCTS (EACH) COMPLETE PRODUCT LINE AND QUALITY IMPROVEMENT (EACH) FINANCIAL PROFITABILITY CAPITAL INVESTMENT REQUIRED ANNUAL (OR UNIT) COST RATE OF RETURN ON INVESTMENT UNIT PRICE PAYOUT PERIOD UTILIZATION OF ASSETS, COST TREND, COST REDUCTION, AND CASH FLOW (EACH) 23 15 9 6 4 3 3 17 10 7 25 4 3 1 TIMING TIMING OF INTRODUCTION OF NEW PRODUCT EXPECTED PRODUCT SALES LIFE CORPORATE OBJECTIVES FITS IN OVERALL OBJECTIVES AND STRATEGY CORPORATE IMAGE Figure 1. Comparative Analysis of Factors Used To Evaluate R&D Project Proposals in 32 Companies. R&D Project Selection POINTS (0 TO 4) CRITERIA 291 x WEIGHT = SCORE ENGINEERING/R&D • LIKELIHOOD OF TECH. SUCCESS • FITS TECHNICAL CAPABILITY • PATENT POSITION • DEVELOPMENT RISK (FEASABILITY) FINANCIAL • RATE OF RETURN ON INVESTMENT • CAPITAL INVESTMENT REQUIRED • PAYBACK PERIOD MANUFACTURING • FITS MANUFACTURING CAPABILITY • PARTS EASILY FABRICATED • RAW MATERIAL AVAILABILITY MARKETING SALES • SIZE OF POTENTIAL MARKET • FITS PRESENT DIST. STRUCTURE • MARKET TREND & GROWTH • CUSTOMER ACCEPTANCE CORPORATE OBJECTIVES • FITS INTO OVERALL STRATEGY • PROPRIETARY POSITION PROJECT TITLE: OBJECTIVES: TOTAL SCORE: SCORED BY: DEPARTMENT: DATE" Figure 2. A Scoring Model for Project Selection. PROJECT SELECTION METHODS Scoring models are perhaps the oldest and most familiar class of models which address the multiple-criteria nature of project selection. Requirements include developing a list of evaluation criteria, reaching a consensus on the weights given to each, and then scoring each project with respect to each of the criteria. A weighted average score is then computed and used in project ranking and selection. A list of often-used criteria is given as Figure 1, while a simple example of such a model is given as Figure 2. A shortcoming o f this approach is the arbitrary choice o f weights for the criteria and the lack of a reliable, consistent scale of measurement across the criteria. Both binary(0-1) integer and non-linear goal programming(GP) recently have been applied to R&D project selection. Their major improvement over standard 0-1 integer programming techniques is the ability to consider several criteria within the objective function. An example summarizing the use of GP is given as Figure 3. ~ However, GP is a m o d e l and not a p r o c e s s , and provides no methods for insuring that the goals selected adequately reflect the organizational and environmental factors related to the project selection decision. Perhaps, a computer-assisted goal selection process, a database on R&D costs and benefits, and GP can be combined into an expert support system for project selection. Multi-attribute utility(MAU) theory can be utilized to model the unique preferences of the decision maker using utility functions which are derived in a specific organizational context. Several applications of MAU to the R&D portfolio investment problem 292 Matthew J. Liberatore in both the private and public sectors have been reported, such as the example s u m m a rized in Figure 4.' Again computer-assisted support in developing the utility functions and attributes to be measured, as well as access to needed databases is required to move toward an expert support system. Some progress has been made in this area. 7 THE ANALYTIC HIERARCHY PROCESS Background. The Analytic Hierarchy Process ( A H P ) allows decision-makers to visually structure a complex problem in the form of a hierarchy having at least two levels: objectives (criteria for evaluation) and activities (products, courses of action, etc.). Each factor or alternative on a given level can be identified and evaluated with respect to other related factors. An important advantage o f A H P is its simplicity. In comparing five conceptually different approaches for determining weights in utility models, it was found that subjects perceived A H P as the easiest method and the one whose results were most trustworthy2 A H P also allows the measurement of inconsistency of h u m a n judgments; if it exceeds a specific limit, some revision of judgment may be required. See Saaty ' for more details on A H P methodology. The model structure presented below reflects the corporate and R&D planning processes in several firms, a m o n g them a diversified chemical manufacturer. DECISION VARIABLES: 0, if project is not accepted X,--1, if project is accepted d- = negative slack deviation variable (underachievement of goal) d * = positive slack deviation variable (overachievement of goal) GOALS (IN ORDER OF PRIORITY)/DEVlATION VARIABLES P,: R&D BUDGET GOAL (d,*) P2: PHYSICAL FACILITIES GOAL (d2*, d3*) P3: MAXIMUM MANPOWER GOAL (d;, d~ P,: PRIORITY PROJECTS GOAL (d~) Ps: OFFENSE-DEFENSE PROJECT BALANCE GOAL (d;, ds~ P6: RISK SPREADING GOAL (dg-) PT: SALES GOAL (d,-o, d,-,) P,: MARKET SHARE GROWTH GOALS (d,-~, d,-3) Pg: MAXIMIZATION OF NET PRESENT VALUES (d;,) PROBLEM MINIMIZE GOAL-WEIGHTED SUM OF DEVIATIONAL VARIABLES SUBJECT TO CONSTRAINTS ON THE ACHIEVEMENT OF EACH GOAL *A. J. Keown, B. W. Taylor, C. P. Duncan, "Allocation of Research and Development Funds: A Zero-One Goal Programming Approach," Omega,Vol. 7. No. 4, pp. 345-351. Figure 3. Zero-One Goal Programming.* R&D Project Selection 293 f, Max Z = _.1Au(a)f(alx)da X s.t. ~x,j_< Bj for all j (budgetary constraints) i and x,j = 0 or L,~~ x,j _<U,j for all i, j (funding ranges) where x,j is the amount of jth resource allocated to the ith project A is the set of n attributes (sales, ROI, profit) KEY ASSUMPTIONS --f(a[x) is a normal density function --u(a) is exponentially constant risk averse utility function SOLUTION APPROACH Three linear approximation formulas are used and compared *G. F. Madey and B. V. Dean, "Strategic Planning in R&D Using Decision Analysis and Mathematical Programming," IEEE Transactions on Engineering Management, Vol. EM-52, Nov. 2, (1979), pp. 84-90. Figure 4. Multi-Attribute Utility M o d e l . " Project Priorities. The first segment of the AHP modeling framework is presented as Figure 5. The scenarios which can best achieve the focus of the hierarchy, called the future of the firm, are developed during the strategic planning process. The three scenarios listed, namely, maintain, expand and diversify business, are often used as corporate planning scenarios. The pairwise comparisons of these scenarios in relation to the focus are provided by the corporate planning committee, which includes the president and area vice presidents. Next, upper R&D management ranks the major R&D "actors" such as process, product and exploratory R&D. The ability of the R&D actors in helping to achieve each scenario will differ. This information can be crucial if, for example, the firm is deciding to place more emphasis on business diversification than in the past. The priorities established thus far (as shown by the weights given at level three of the hierarchy) provide the linkage between the strategy of the firm and the general direction of R&D efforts. A major drawback of many R&D project selection processes is that this top-level input is never clearly revealed to, or understood by, R&D during the planning cycle. The results can be unclear priorities and constant revision of the project investment portfolio. The next stage of the modeling process, as shown in Figure 6, requires the directors of each of the major R&D activities (actors) to develop criteria and subcriteria for project proposal evaluation. The specific criteria may differ for each R&D group, and the final set should to be acceptable to both the R&D director and his superior. Four categories are listed for each R&D actor, and include technical, marketing/distribution, manufac- 294 Matthew J. Liberatore FUTURE OF THE FIRM LEVEL I: FOCUS LEVEL If: SCENARIOS LEVEL I I I : ACTORS MAINTAIN EX ISTI NG BUSINESSES .625 EXPAND EXISTING BUSINESSES .238 PROCESS I R&D .646 PRODUCT R&D .193 DIVERSIFY INTO NEW BUSINESSES .136 Figure 5. Top levels of AHP modeling framework. turing, financial, and general. Pairwise comparison matrices must be developed to reflect the importance of these criteria in relation to the three R&D actors on an individual basis. Within each criteria, subcriteria must be established and again pairwise compared to determine their priority within the overall criteria. If the number of projects within each R&D activity area is small, generally seven or less, the projects can be pairwise compared with respect to each subcriteria, as indicated in Figure 6. However, when the number of projects is large, such methods are generally infeasible. For example, if there are 50 process R&D projects then, n ( n - 1)/2= 1225 pairwise comparisons are required for each of the twelve subcriteria listed. The explosion in the number of required comparisons is a criticism of the basic A H P approach. Fortunately, other methods are available to reduce the number of required judgments. One approach' is illustrated in Figure 7. ~ For each subcriterion a series of performance ratings are established and weights or priorities are determined for each. These rating levels are coded as follows: outstanding, above average, average and below average. For each subcriterion, say manufacturing capability, pairwise comparisons between the four rating levels are required. For example, in evaluating process R&D projects, how much more important is an outstanding rating than an above average rating for manufacturing capability? These comparisons lead to priorities or weights for each of the four ratings levels associated with each subcriterion. These weights are then scaled across the R&D project selection hierarchy to determine a final weight for each rating level by subcriteria. The resulting weights and their associated criteria then can be transferred to a spreadsheet program so that a rating for each subcriteria can be R&D Project Selection 295 selected for each project. The weights for the selected ratings are added for a total project score and then renormalized to sum to one. The project rating spreadsheet for process R&D is given as Table 1. The rating spreadsheets for products and exploratory R&D would be similar. Table 2 summarizes the results of the ratings analysis for the 27 hypothetical projects used in this example. The development and analysis presented was accomplished using the microcomputer programs Expert Choice and Lotus 1-2-3 on an IBM PC with 512k o f memory. This approach differs from that o f standard scoring models, since the weights provided for the ratings of each subcriterion are not based on arbitrary scales, but utilize a ratio scale for human judgments. The combined AHP-spreadsheet approach is easy to use and can accommodate inconsistencies in human judgments. Finally, it should be noted that the weights or priorities determined for each of the three actors (process, product and exploratory R&D) as shown in Figure 5 are used to scale the ratings for each subcriterion for each R&D actor. It is this step that provides the linkage between the R&D actors' ability to help achieve the business planning scenarios and the rating of individual projects according to the various subcriteria. Project Resource Allocation. Several methods are available which consider the tradeoffs between project benefits and costs, and in some cases address other factors related to the mix of projects funded. These include benefit-cost analysis and mathematical programming. A straightforward application of benefit-cost analysis requires taking the ratio of the renormalized project priority, representing the sum total of the project's PROCESS R&D LEVEL I l l : ACTOR .646 LEVEL IV: CRITER IA LEVEL V : SUBCRITERIA I I MANUE (.407) CAPABILITY .422 FAC./EQU IPT . 2 8 5 -- I TECHNICAL I MRKT./DISTRIB. ("48----~L)- I (055, I POTENTIAL PROBABILITY SUCCESS .018 ICAPABILITY COSTS .Oel I TRENDS TIME . O.'.W,8 / ,'E "RCES I I .030 I .0t8 I .0071 I 1 1 (035) I FINANCIAL PI PROFITABILITY .023 I CAPITAL INV, .014 l C~ U UN"i "°°"I \V LEVEL Vl: PROJECTS PROJECT4 .27"/' PROJECT 5 I •216 I I PROJECT 6 .153 Figure 6. Lower levels of AHP modeling framework for a small number of project alternatives. 296 Matthew J. Liberatore LEVEL III : ACTORS // LEVEL IV: CRITERIA I Manufacturing .407 [ LEVEL V: SUBCRITERIA LEVEL Vl: RATING SCALE Process R~D .646 Capability .122 I Outstanding .067 I Facilities/ Equipment .285 1 Averoge .016 Above Avg. .032 I Outstanding .158 Below Avg. .005 Above Avg. .076 I Average .038 Below Avg. .013 LEVEL V II : PROJECTS Figure 7. Lower levels of AHP modeling framework for a large number of project alternatives. R&D Project S e l e c t i o n 297 Table 1. Partial R&D Ranking Model for Process Projects. PROCESS PROJECTS 1 MNFB CAPABLTY OUTSTAND ABV.AVG AVERAGE BELOW AV FAC/EQPT O UTSTAN D ABV.AVG AVERAGE BELOW AV TECNICAL PRBL SCS OUTSTAND ABV.AVG. AVERAGE BELOW AV COSTS OUTSTAND ABV.AVG AVERAGE BELOW AV TIME OUTSTAND ABV.AVG AVERAGE BELOW AV RESOURCS OUTSTAND ABV.AVG AVERAGE BELOW AV MRKT/DST POTNTIAL OUTSTAND ABV.AVG AVERAGE BELOW AV CAPABLTY OUTSTAND ABV.AVG AVERAGE BELOW AV TRENDS OUTSTAND ABV.AVG AVERAGE BELOW AV FIN PRFTBLY OUTSTAND ABV.AVG AVERAGE BELOW AV CAP INV OUTSTAND ABV.AVG 2 3 0.067717 0 0 0 0 0 0 0.005676 0 0.03262 0 0 0 0.076113 0 0 0.158006 0 0 0 0 0.076113 0 0 0 0 0.002312 0 0.009597 0 0 0 0 0 0.002312 0 0 0.021750 0 0 0 0 0.010878 0 0 0.021750 0 0 0 0 0 0.001814 0 0 0 0.001814 0 0 0 0.001814 0 0.002669 0 0 0 0 0 0.000464 0.005541 0 0 0 0 0.008126 0 0 0 0 0 0,001414 0 0 0.004064 0 0 0 0.002326 0 0 0 0.002326 0 0 0 0 0.000809 0.003684 0 0 0 0 0.001774 0 0 0 0 0.000887 0 0 0 0.003131 0 0 0.006260 0 0 0 0.006260 0 0 0 0 0 0.002497 0 0.002497 O.645992 0.407759 0,122328 0.067717 0.03262 0.016314 0.005676 0.285431 0.158006 0.076113 0.038066 0.013246 0.148014 0.017336 0.009597 0.004623 0.002312 0.000804 0.081567 0.045153 0.021750 0.010878 0.003785 0.039099 0.021644 0.010426 0.005214 0.001814 0.010010 0.005541 0.002669 0.001335 0.000464 0.054570 0.030473 0.016869 0.008126 0.004064 0.001414 0.017441 0.009655 0.004651 0.002326 0.000809 0.006655 0.003684 0.001774 0.000887 0.000308 0.035647 0.023478 0.012997 0.006260 0.003131 0.001089 0.009366 0.005185 0.002497 (continued) 298 M a t t h e w J. L i b e r a t o r e Table 1. Continued. PROCESS PROJECTS 1 AVERAGE BELOW AV UNIT CST OUTSTAND ABV.AVG AVERAGE BELOW AV 2 3 0.001249 0 0 0 0 0 0 0 0 0.00013 0 0 0.000373 0 0 0 0.000373 0 0.191023 0.201084 0.155045 0.001249 0.000434 0.002802 0.001551 0.000747 0.000373 0.00013 Table 2. Resource Allocation Using Priority-Cost Analysis and Integer Programming. PROJECTS SCORE 14 12 18 13 16 4 10 1 17 15 24 11 2 20 23 22 9 7 21 27 8 6 5 19 3 26 25 0.172763 0.099258 0.121581 0.149583 0.088454 0.056544 0.122882 0.059700 0.086136 0.149258 0.028952 0.127774 0.062159 0.048981 0.049969 0.029636 0.033721 0.063494 0.048694 0.043945 0.051993 0.022061 0.044219 0.036498 0.043655 0.033558 0.029082 RENORM. SCORE PROJECT COST (000) PRIORITYCOST* CUM COST 0.090710 0.052116 0.063836 0.078539 0.046443 0.029689 0.064520 0.031346 0.045226 0.078368 0.015201 0.067088 0.032637 0.025717 0.026236 0.015560 0.017705 0.033337 0.025567 0.023073 0.027299 0.011583 0.023217 0.019163 0.022921 0.017619 0.015269 1 30 20 25 50 30 20 45 25 40 70 15 80 40 35 40 25 30 60 50 55 70 30 65 60 90 75 95 :270 3.023683 2.605815 2.553473 1.570788 1.548113 1.484456 1.433783 1.253841 1.130654 1.119552 1.013432 0.838606 0.815929 0.734791 0.655922 0.622437 0.590188 0.555632 0.511348 0.419526 0.389988 0.386118 0.357197 0.319397 0.254683 0.234932 0.160734 30 50 75 125 155 175 220 245 285 355 370 450 490 525 565 590 620 680 730 785 855 885 950 1010 1100 1175 1270 * Rescaled as a number between 0 and 10. ° "1 -- Additional Projects Funded if B = $500. 2 = Additional Projects Funded if B -- $750. 3 = Additional Projects Funded if B = $1000. IP CODE* * 1 1 1 1 1 1 1 1 1 1 2 1 1 2 2 1 2 2 3 3 2 3 3 R&D Project Selection 299 benefits, to the cost o f funding the project. The results of benefit-cost analysis for our example are also given as Table 2. The projects can be selected in descending order o f their priority-cost ratio until the R&D budget is depleted. A second general approach is the use of 0-1 integer linear programming. The problem objective is to maximize total priority over all funded projects, subject to a budgetary constraint (B) and possibly other restrictions. A variety of additional constraints can be added to insure certain characteristics in the mix of total projects funded. For example, constraints can require a m i n i m u m or m a x i m u m number of projects to be funded in each of the three basic R&D areas (process, product and exploratory). The example problem was run on the L I N D O program using a single budgetary constraint at three funding levels: $500, $750 and $1000. The results are also summarized in Table 2, which allows comparisons with the priority-cost analysis. The results of both analyses are nearly identical at the $500 level, with the exception that project 22 instead o f project 24 is funded in the integer programming solution to allow for full utilization of the budget. A similar switch at the $750 budget level occurs with the substitution of project 8 for project 21. SUMMARY AND CONCLUSIONS An expert support system for R&D project selection must consider: 1) the organizational context o f the R&D occurs; 2) economic as well as social benefit-cost analysis; and 3) measurement and aggregation of multiple selection criteria. Four multiple-criteria decision-making methods were reviewed and examples presented: scoring models, goal programming, multiattribute utility theory, and the analytic hierarchy process. An extended example is presented to illustrate the development o f a combined A H P and spreadsheet expert support system for project prioritization and resource allocation. Scoring models lack a consistent, reliable measurement scale, and so are probably not suited as a c o m p o n e n t in an expert support system. G P and MAU can be incorporated into an expert support system, but must be linked to additional components which enable the expert to specify the criteria and which simplify measurement problems. They must also allow for rapid reevaluation of project priorities as information and opinions change. The latter probably indicates that more heuristics must be developed to ease the computational burden of these methods. The usage of microcomputer-based software such as Expert Choice and Lotus 1-2-3 can form the basis o f an expert support system for R&D project management. Additional software is required to move the resulting system closer to an expert system. For example, computer-support could be used to assist in the development of planning scenarios, selection criteria and project alternatives; and previous judgments by decision-makers concerning the pairwise comparison of certain objective data (such as financial measures) could be used to reduce the time and effort required for the collection of pairwise comparison data. REFERENCES 1. Baker, Norman R., R&D project selection models: an assessment, IEEE Trans. Eng. Manag., Vol. EM-21, No. 4 165-171, (1974). 2. Baker, Norman R. and Pound, W. H. R&D project selection: where we stand, IEEE Trans. Eng. Manag.o Vol. EM-I I, No. 4: 124-134, (1964). 3. Forman, Ernest H., Saaty, Thomas L., Selly, Mary Ann, and Waldron, Rozann, Expert Choice, Decision Support Software, McLean, VA, (1983). 4. Gupta, Sushil K. and Taube, Larry R., State of the art survey on project management, in Project Management Methods and Studies, B. V. Dean (ed.), North-Holland, Amsterdam, (1985). 300 Matthew J. Liberatore 5. Keown, A. J., Taylor, B. W., and Duncan, C. P., Allocation of research and development funds: a zero-one goal programming approach, Omega, Vol. 7: 345-351, (1979). 6. Liberatore, Matthew, J., and Titus, George J., The practice of management science in R&D project management, Management Sci., Vol. 29, No. 8: 962-974, (1983). 7. Madey, Gregory, R., and Dean, Burton V., Strategic planning for investment in R&D using decision analysis and mathematical programming, IEEE Trans. Eng. Manag., Vol. EM-32, No. 2: 84-90, (1985). 8. Saaty, Thomas, The Analytic Hierarchy Process, McGraw-Hill, New York, (1980). 9. Schoemaker, Paul J. H., and Carter, C. Waid, An experimental comparison of different approaches to determining weights in additive utility models, Management Sci., Vol. 28, No. 2: 182-196, (1982).