research-article

Open access

Exact scalable sensitivity analysis for the next release problem

Authors:

Mark Harman,

Jens Krinke,

Inmaculada Medina-Bulo,

Francisco Palomo-Lozano,

Jian Ren,

Shin YooAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 23, Issue 2

Article No.: 19, Pages 1 - 31

https://doi.org/10.1145/2537853

Published: 04 April 2014 Publication History

PDF eReader

Abstract

The nature of the requirements analysis problem, based as it is on uncertain and often inaccurate estimates of costs and effort, makes sensitivity analysis important. Sensitivity analysis allows the decision maker to identify those requirements and budgets that are particularly sensitive to misestimation. However, finding scalable sensitivity analysis techniques is not easy because the underlying optimization problem is NP-hard. This article introduces an approach to sensitivity analysis based on exact optimization. We implemented this approach as a tool, OATSAC, which allowed us to experimentally evaluate the scalability and applicability of Requirements Sensitivity Analysis (RSA). Our results show that OATSAC scales sufficiently well for practical applications in Requirements Sensitivity Analysis. We also show how the sensitivity analysis can yield insights into difficult and otherwise obscure interactions between budgets, requirements costs, and estimate inaccuracies using a real-world case study.

References

[1]

E. Alba and F. Chicano. 2007. Software project management with GAs. Inf. Sci. 177, 11, 2380--2401.

Digital Library

Google Scholar

[2]

T. Albourae, G. Ruhe, and M. Moussavi. 2006. Lightweight replanning of software product releases. In Proceedings of the 1st International Workshop on Software Product Management (IWSPM'06). IEEE Computer Society, 27--34.

Digital Library

Google Scholar

[3]

R. Alvarez-Valdés, E. Crespo, J. M. Tamarit, and F. Villa. 2006. A scatter search algorithm for project scheduling under partially renewable resources. J. Heuristics 12, 1--2, 95--113.

Digital Library

Google Scholar

[4]

G. Antoniol, M. Di Penta, and M. Harman. 2004. A robust search-based approach to project management in the presence of abandonment, rework, error and uncertainty. In Proceedings of the 10th International Symposium on the Software Metrics (METRICS'04). IEEE Computer Society, 172--183.

Digital Library

Google Scholar

[5]

G. Antoniol, M. D. Penta, and M. Harman. 2005. Search-based techniques applied to optimization of project planning for a massive maintenance project. In Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM'05). IEEE Computer Society, Los Alamitos, CA, 240--249.

Digital Library

Google Scholar

[6]

A. Bagnall, V. Rayward-Smith, and I. Whittley. 2001. The next release problem. Inf. Softw. Technol. 43, 14, 883--890.

Crossref

Google Scholar

[7]

P. Baker, M. Harman, K. Steinhofel, and A. Skaliotis. 2006. Search based approaches to component selection and prioritization for the next release problem. In Proceedings of the 22nd IEEE International Conference on Software Maintenance (ICSM'06). IEEE Computer Society, 176--185.

Digital Library

Google Scholar

[8]

R. Beier and B. Vöcking. 2004. Random knapsack in expected polynomial time. J. Comput. Syst. Sci. 69, 3, 306--329.

Digital Library

Google Scholar

[9]

R. E. Bixby, M. Fenelon, and Z. Gu. 2000. MIP: Theory and practice—closing the gap. Syst. Model. Optim.: Methods, Theory, Appl. 1, 1960, 1--31.

Google Scholar

[10]

C. Chao, J. Komada, Q. Liu, M. Muteja, Y. Alsalqan, and C. Chang 1993. An application of genetic algorithms to software project management. In Proceedings of the 9th International Advanced Science and Technology. 247--252.

Google Scholar

[11]

B. H. Cheng and J. M. Atlee. 2007. Research directions in requirements engineering. In Proceedings of the Future of Software Engineering. IEEE Computer Society, Los Alamitos, CA, 285--303.

Digital Library

Google Scholar

[12]

T. E. Colanzi, S. R. Vergilio, W. K. G. Assuno, and A. Pozo. 2012. Search based software engineering: Review and analysis of the field in Brazil. J. Syst. Softw.

Digital Library

Google Scholar

[13]

J. T. de Souza, C. L. Maia, F. G. de Freitas, and D. P. Coutinho. 2010. The human competitiveness of search based software engineering. In Proceedings of the 2nd International Symposium on Search Based Software Engineering (SSBSE'10). IEEE Computer Society.

Digital Library

Google Scholar

[14]

M. S. Feather, S. L. Cornford, J. D. Kiper, and T. Menzies. 2006. Experiences using visualization techniques to present requirements, risks to them, and options for risk mitigation. In Proceedings of the International Workshop on Requirements Engineering Visualization (REV'06). IEEE, 10--10.

Digital Library

Google Scholar

[15]

M. S. Feather, J. D. Kiper, and S. Kalafat. 2004. Combining heuristic search, visualization and data mining for exploration of system design space. In Proceedings of the 14th Annual International Symposium of International Council on Systems Engineering (INCOSE'04).

Google Scholar

[16]

M. S. Feather and T. Menzies. 2002. Converging on the optimal attainment of requirements. In Proceedings of the 10th IEEE International Conference on Requirements Engineering (RE'02). IEEE, 263--270.

Digital Library

Google Scholar

[17]

A. Finkelstein, M. Harman, S. A. Mansouri, J. Ren, and Y. Zhang. 2008a. Fairness analysis in requirements assignments. In Proceedings of the 16th IEEE International Requirements Engineering Conference (RE'08). IEEE Computer Society, 115--124.

Digital Library

Google Scholar

[18]

A. Finkelstein, M. Harman, S. A. Mansouri, J. Ren, and Y. Zhang. 2008b. Fairness analysis in requirements assignments. In Proceedings of the 16th IEEE International Requirements Engineering Conference (RE'08).

Digital Library

Google Scholar

[19]

F. G. Freitas and J. T. Souza. 2011. Ten years of search based software engineering: A bibliometric analysis. In Proceedings of the 3rd International Symposium on Search Based Software Engineering (SSBSE'11). 18--32.

Digital Library

Google Scholar

[20]

H. C. Frey and S. Patil. 2002. Identification and review of sensitivity analysis methods. Risk Anal. 22, 3, 553--578.

Crossref

Google Scholar

[21]

M. R. Garey and D. S. Johnson. 1978. Strong NP-completeness results: Motivation, examples, and implications. J. ACM 25, 499--508.

Digital Library

Google Scholar

[22]

M. R. Garey and D. S. Johnson. 1979. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman.

Digital Library

Google Scholar

[23]

D. Greer and G. Ruhe. 2004. Software release planning: An evolutionary and iterative approach. Inf. Softw. Technol. 46, 4, 243--253.

Crossref

Google Scholar

[24]

D. M. Hamby. 1994. A review of techniques for parameter sensitivity analysis of environmental models. Environ. Monitor. Assess. 32, 2, 135--154.

Crossref

Google Scholar

[25]

M. Harman. 2007. The current state and future of search based software engineering. In Proceedings of the Future of Software Engineering, L. Briand and A. Wolf, Eds. IEEE Computer Society Press, Los Alamitos, CA, 342--357.

Digital Library

Google Scholar

[26]

M. Harman, E. Burke, J. A. Clark, and X. Yao. 2012a. Dynamic adaptive search based software engineering. In Proceedings of the 6th IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM'12).

Digital Library

Google Scholar

[27]

M. Harman, J. Krinke, J. Ren, and S. Yoo. 2009. Search based data sensitivity analysis applied to requirement engineering. In Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation (GECCO'09). ACM, 1681--1688.

Digital Library

Google Scholar

[28]

M. Harman, A. Mansouri, and Y. Zhang. 2012b. Search based software engineering: Trends, techniques and applications. ACM Comput. Surv. 45, 1, Article 11.

Digital Library

Google Scholar

[29]

M. Harman, P. Mcminn, J. Souza, and S. Yoo. 2012c. Search based software engineering: Techniques, taxonomy, tutorial. In Empirical Software Engineering and Verification: (LASER'09--10), B. Meyer and M. Nordio, Eds., Lecture Notes in Computer Science, vol. 7007, Springer, 1--59.

Digital Library

Google Scholar

[30]

J. Helton, J. Johnson, C. Sallaberry, and C. Storlie. 2006. Survey of sampling-based methods for uncertainty and sensitivity analysis. Reliability Eng. Syst. Safety 91, 10--11, 1175--1209.

Crossref

Google Scholar

[31]

O. Jalali, T. Menzies, and M. Feather. 2008. Optimizing requirements decisions with KEYS. In Proceedings of the 4th International Workshop on Predictor Models in Software Engineering (PROMISE'08). ACM, 79--86.

Digital Library

Google Scholar

[32]

P. Kapur, A. Ngo-The, G. Ruhe, and A. Smith. 2008. Optimized staffing for product releases and its application at Chartwell Technology. J. Softw. Maintenance Evol. Res. Pract. (Special Issue Search Based Software Engineering), 20, 5, 365--386.

Digital Library

Google Scholar

[33]

J. Karlsson, C. Wohlin, and B. Regnell. 1998. An evaluation of methods for priorizing software requirements. Info. Softw. Technol. 39, 14--15, 939--947.

Crossref

Google Scholar

[34]

R. Karp. 1972. Reducibility among combinatorial problems. Complex. Comput. Compu. 40, 4, 85--103.

Crossref

Google Scholar

[35]

R. Levine and D. Renelt. 1992. A sensitivity analysis of cross-country growth regressions. Amer. Econ. Rev. 82, 4, 942--963.

Google Scholar

[36]

C. Li, M. van den Akker, S. Brinkkemper, and G. Diepen. 2007. Integrated requirement selection and scheduling for the release planning of a software product. In Proceedings of the 13th International Working Conference on Requirements Engineering: Foundation for Software Quality (RefsQ'07). Lecture Notes in Computer Science, vol. 4542. Springer, 93--108.

Digital Library

Google Scholar

[37]

G. L. Nemhauser and Z. Ullmann. 1969. Discrete dynamic programming and capital allocation. Manage. Sci. 15, 9, 494--505.

Digital Library

Google Scholar

[38]

J. Newman, A. Taylor, R. Barnwell, and P. Newman. 1999. Overview of sensitivity analysis and shape optimization for complex aerodynamic configurations. J. Aircraft 36, 1, 87--96.

Crossref

Google Scholar

[39]

A. Ngo-The and G. Ruhe. 2009. Optimized resource allocation for software release planning. IEEE Trans. Softw. Eng. 35, 1, 109--123.

Digital Library

Google Scholar

[40]

D. Pisinger. 2005. Where are the hard knapsack problems&quest; Comput. Oper. Res. 32, 9, 2271--2284.

Digital Library

Google Scholar

[41]

R. Racu, M. Jersak, and R. Ernst. 2005. Applying sensitivity analysis in real-time distributed systems. In Proceedings of the 11th IEEE Real Time and Embedded Technology and Applications Symposium. IEEE Computer Society, 160--169.

Digital Library

Google Scholar

[42]

G. N. Rodrigues, D. S. Rosenblum, and S. Uchitel. 2005. Sensitivity analysis for a scenario-based reliability prediction model. In Proceedings of the Workshop on Architecting Dependable Systems (WADS'05). ACM, New York, NY, 1--5.

Digital Library

Google Scholar

[43]

G. Ruhe, A. Eberlein, and D. Pfahl. 2003. Trade-off analysis for requirements selection. Int. J. Softw. Eng. Knowl. Eng. 13, 4, 345--366.

Crossref

Google Scholar

[44]

G. Ruhe and D. Greer. 2003. Quantitative studies in software release planning under risk and resource constraints. In Proceedings of the International Symposium on Empirical Software Engineering (ISESE'03). IEEE, 262--270.

Digital Library

Google Scholar

[45]

G. Ruhe and A. Ngo-The. 2004. Hybrid intelligence in software release planning. Int. J. Hybrid Intell. Syst. 1, 1--2, 99--110.

Digital Library

Google Scholar

[46]

M. O. Saliu and G. Ruhe. 2007a. Bi-objective release planning for evolving software systems. In Proceedings of the the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC-FSE'07). ACM, New York, NY, 105--114.

Digital Library

Google Scholar

[47]

M. O. Saliu and G. Ruhe. 2007b. Bi-objective release planning for evolving software systems. In Proceedings of the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE'07). I. Crnkovic and A. Bertolino, Eds., ACM, 105--114.

Digital Library

Google Scholar

[48]

A. Saltelli and P. Annoni. 2010. How to avoid a perfunctory sensitivity analysis. Environ. Model. Softw. 25, 12, 1508--1517.

Digital Library

Google Scholar

[49]

A. Saltelli, M. Ratto, T. Andres, F. Campolongo, J. Cariboni, D. Gatelli, M. Saisana, and S. Tarantola. 2008. Global Sensitivity Analysis: The Primer. John Wiley & Sons, Sussex.

Google Scholar

[50]

A. Saltelli, S. Tarantola, and F. Campolongo. 2000. Sensitivity analysis as an ingredient of modeling. Stat. Sci. 15, 4, 377--395.

Crossref

Google Scholar

[51]

A. Sandu, D. N. Daescu, and G. R. Carmichael. 2003. Direct and adjoint sensitivity analysis of chemical kinetic systems with KPP: Part I--theory and software tools. Atmospheric Environ. 37, 36, 5083--5096.

Crossref

Google Scholar

[52]

M. Shepperd. 2007. Software project economics: A roadmap. In Proceedings of the Future of Software Engineering (FoSE'07). IEEE Computer Society, 304--315.

Digital Library

Google Scholar

[53]

M. van den Akker, S. Brinkkemper, G. Diepen, and J. Versendaal. 2004. Flexible release composition using integer linear programming. Tech. rep. UU-CS-2004-063, Institute of Information and Computing Sciences, Utrecht University.

Google Scholar

[54]

M. van den Akker, S. Brinkkemper, G. Diepen, and J. Versendaal. 2005. Flexible release planning using integer linear programming. In Proceedings of the 11th International Workshop on Requirements Engineering for Software Quality (RefsQ'05). Essener Informatik Beitrage, 247--262.

Google Scholar

[55]

S. Wagner. 2007a. An approach to global sensitivity analysis: FAST on COCOMO. In Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM'07). IEEE Computer Society, Los Alamitos, CA, 440--442.

Digital Library

Google Scholar

[56]

S. Wagner. 2007b. Global sensitivity analysis of predictor models in software engineering. In Proceedings of the 3rd International Workshop on Predictor Models in Software Engineering (PROMISE'07). IEEE Computer Society, 3--10.

Digital Library

Google Scholar

[57]

Y. Zhang, A. Finkelstein, and M. Harman. 2008. Search based requirements optimisation: Existing work and challenges. In Proceedings of the International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ'08). Lecture Notes in Computer Science, vol. 5025, Springer, 88--94.

Digital Library

Google Scholar

[58]

Y. Zhang, M. Harman, and S. A. Mansouri. 2007. The multi-objective next release problem. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM Press, New York, NY, 1129--1136.

Digital Library

Google Scholar

[59]

L. Zhu, A. Aurum, I. Gorton, and R. Jeffery. 2005. Tradeoff and sensitivity analysis in software architecture evaluation using analytic hierarchy process. Softw. Quality Control 13, 4, 357--375.

Digital Library

Google Scholar

Cited By

View all

Ghasemi MBagherifard KParvin HNejatian S(2023)Novel multi objective evolutionary framework for solving next release problemJournal of Intelligent & Fuzzy Systems10.3233/JIFS-20022344:2(3315-3339)Online publication date: 30-Jan-2023
https://doi.org/10.3233/JIFS-200223
Massobrio R(2022)Learning for Optimization with Virtual SavantMemoria Investigaciones en Ingeniería10.36561/ING.22.4(29-39)Online publication date: 30-Jun-2022
https://doi.org/10.36561/ING.22.4
Dukhan WMohamed MAmer AZanaty EReyad O(2022)Software requirement selection using a combined multi‐objective optimisation techniqueIET Software10.1049/sfw2.1207016:6(558-575)Online publication date: 12-Aug-2022
https://doi.org/10.1049/sfw2.12070
Show More Cited By

Index Terms

Exact scalable sensitivity analysis for the next release problem
1. Software and its engineering
  1. Software creation and management
    1. Designing software
      1. Requirements analysis

Recommendations

Sensitivity and uncertainty analysis for the DeNitrification-DeComposition model, a case study of modeling soil organic carbon dynamics at a long-term observation site with a rice-bean rotation

Long-term global sensitivity analyses have been made for DNDC modeling SOC.Six different uncertainty methods are compared for their suitability.Morris produced similar sensitivity results with fewer DNDC runs compared with Sobol'.Morris combined a user ...
Random network models and sensitivity algorithms for the analysis of ordering time and inventory state in multi-stage supply chains

Supply chains in reality face a highly dynamic and uncertain environment, especially the uncertain end-customer demands and orders. Since the condition of product market changes frequently, the tasks of order management, product planning, and inventory ...
Sensitivity analysis of spatial models

Sensitivity analysis is the study of how uncertainty in model predictions is determined by uncertainty in model inputs. A global sensitivity analysis considers the potential effects from the simultaneous variation of model inputs over their finite range ...

Reviews

Reviewer: Richard John Botting

The next release problem (NRP) is part of the search-based software engineering (SBSE) paradigm. SBSE studies algorithms that find solutions to software developers' problems. The original NRP (2001) [1] was to select the best set of requirements to tackle in the next release of a product. Given estimates of costs and revenues, the objective is to maximize revenue within budget. This paper studies the special case of independent requirements. It shows how to discover costs, revenues, and budgets that have a big effect on the outcome. The paper shows that the Nemhauser-Ullmann algorithm has the predicted polynomial mean time and is fast enough on practical examples to be run many times to discover sensitive cases. The authors note that pathological cases lead to the exponential worst cases of an NP-hard problem. They do not say that using randomization handles such problems in practice. They test their algorithm on the Motorola dataset, but give no source. In a software project, the problem of choosing the next requirements to be implemented is more complex than the NRP. The best strategy does not optimize revenue, but targets risky or novel features and architectures first. The early iterations of a project make sure the estimates are solid and no architectural surprises will derail it. Only in later iterations does the NRP apply. This paper advances academic research in SBSE, but provides a snare and delusion for software developers and their managers. Online Computing Reviews Service

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 23, Issue 2

March 2014

319 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/2600788

Editor:
David S. Rosenblum

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 April 2014

Accepted: 01 October 2013

Revised: 01 April 2013

Received: 01 May 2012

Published in TOSEM Volume 23, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
1,054
Total Downloads

Downloads (Last 12 months)61
Downloads (Last 6 weeks)8

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Ghasemi MBagherifard KParvin HNejatian S(2023)Novel multi objective evolutionary framework for solving next release problemJournal of Intelligent & Fuzzy Systems10.3233/JIFS-20022344:2(3315-3339)Online publication date: 30-Jan-2023
https://doi.org/10.3233/JIFS-200223
Massobrio R(2022)Learning for Optimization with Virtual SavantMemoria Investigaciones en Ingeniería10.36561/ING.22.4(29-39)Online publication date: 30-Jun-2022
https://doi.org/10.36561/ING.22.4
Dukhan WMohamed MAmer AZanaty EReyad O(2022)Software requirement selection using a combined multi‐objective optimisation techniqueIET Software10.1049/sfw2.1207016:6(558-575)Online publication date: 12-Aug-2022
https://doi.org/10.1049/sfw2.12070
Marghny MZanaty EDukhan WReyad O(2022)A hybrid multi-objective optimization algorithm for software requirement problemAlexandria Engineering Journal10.1016/j.aej.2021.12.04361:9(6991-7005)Online publication date: Sep-2022
https://doi.org/10.1016/j.aej.2021.12.043
Massobrio RNesmachnow SPalomo-Lozano FDorronsoro B(2021)Virtual Savant as a generic learning approach applied to the basic independent Next Release ProblemApplied Soft Computing10.1016/j.asoc.2021.107374108(107374)Online publication date: Sep-2021
https://doi.org/10.1016/j.asoc.2021.107374
Sadiq MParveen AJain S(2021)Software Requirements Selection with Incomplete Linguistic Preference RelationsBusiness & Information Systems Engineering10.1007/s12599-021-00696-xOnline publication date: 27-Apr-2021
https://doi.org/10.1007/s12599-021-00696-x
Ghasemi MBagherifard KParvin HNejatian SPho K(2021)Multi-objective whale optimization algorithm and multi-objective grey wolf optimizer for solving next release problem with developing fairness and uncertainty quality indicatorsApplied Intelligence10.1007/s10489-020-02018-251:8(5358-5387)Online publication date: 8-Jan-2021
https://dl.acm.org/doi/10.1007/s10489-020-02018-2
Mougouei DPowers D(2020)Dependency-Aware Software Requirements Selection using Fuzzy Graphs and Integer ProgrammingExpert Systems with Applications10.1016/j.eswa.2020.113748(113748)Online publication date: Aug-2020
https://doi.org/10.1016/j.eswa.2020.113748
Ramírez ADelgado-Pérez PFerrer JRomero JMedina-Bulo IChicano F(2020)A systematic literature review of the SBSE research community in SpainProgress in Artificial Intelligence10.1007/s13748-020-00205-3Online publication date: 18-Apr-2020
https://doi.org/10.1007/s13748-020-00205-3
del Sagrado Jdel Águila I(2020)Assisted requirements selection by clusteringRequirements Engineering10.1007/s00766-020-00341-1Online publication date: 16-Nov-2020
https://doi.org/10.1007/s00766-020-00341-1
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Index Terms

Recommendations

Sensitivity and uncertainty analysis for the DeNitrification-DeComposition model, a case study of modeling soil organic carbon dynamics at a long-term observation site with a rice-bean rotation

Random network models and sensitivity algorithms for the analysis of ordering time and inventory state in multi-stage supply chains

Sensitivity analysis of spatial models

Reviews

Access critical reviews of Computing literature here

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Get Access

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations