research-article

Departures from optimality: understanding human analyst's information foraging in assisted requirements tracing

Authors:

Gary BradshawAuthors Info & Claims

ICSE '13: Proceedings of the 2013 International Conference on Software Engineering

Pages 572 - 581

Published: 18 May 2013 Publication History

Abstract

Studying human analyst's behavior in automated tracing is a new research thrust. Building on a growing body of work in this area, we offer a novel approach to understanding requirements analyst's information seeking and gathering. We model analysts as predators in pursuit of prey --- the relevant traceability information, and leverage the optimality models to characterize a rational decision process. The behavior of real analysts with that of the optimal information forager is then compared and contrasted. The results show that the analysts' information diets are much wider than the theory's predictions, and their residing in low-profitability information patches is much longer than the optimal residence time. These uncovered discrepancies not only offer concrete insights into the obstacles faced by analysts, but also lead to principled ways to increase practical tool support for overcoming the obstacles.

References

[1]

J. H. Hayes, A. Dekhtyar, and S. K. Sundaram, “Advancing candidate link generation for requirements tracing: the study of methods,” IEEE TSE, vol. 32(1), pp. 4–19, 2006.

Digital Library

[2]

G. Antoniol, G. Canfora, G. Casazza, A. De Lucia, and E. Merlo, “Recovering traceability links between code and documentation,” IEEE TSE, vol. 28(10), pp. 970–983, 2002.

Digital Library

[3]

A. De Lucia, F. Fasano, R. Oliveto, and G. Tortora, “Recovering traceability links in software artifact management systems using information retrieval methods,” ACM TOSEM, vol. 16(4), 2007.

Digital Library

[4]

X. Zou, R. Settimi, and J. Cleland-Huang, “Improving automated requirements trace retrieval: a study of term-based enhancement methods,” Empir Softw Eng, vol. 15(2), pp. 119–146, 2010.

Digital Library

[5]

X. Chen and J. Grundy, “Improving automated documentation to code traceability by combining retrieval techniques,” in ASE, 2011, pp. 223– 232.

Digital Library

[6]

U.S. Federal Aviation Administration, “RTCA/DO-178B Software Considerations in Airborne Systems and Equipment Certification,” 1992.

[7]

D. Cuddeback, A. Dekhtyar, and J. H. Hayes, “Automated requirements traceability: the study of human analysts,” in RE, 2010, pp. 231–240.

Digital Library

[8]

A. Dekhtyar, O. Dekhtyar, J. Holden, J. H. Hayes, D. Cuddeback, and W.-K. Kong, “On human analyst performance in assisted requirements tracing: statistical analysis,” in RE, 2011, pp. 111–120.

Digital Library

[9]

J. H. Hayes and A. Dekhtyar, “Humans in the traceability loop: can’t live with ’em, can’t live without ’em,” in TEFSE, 2005, pp. 20–23.

Digital Library

[10]

W.-K. Kong, J. H. Hayes, A. Dekhtyar, and J. Holden, “How do we trace requirements? an initial study of analyst behavior in trace validation tasks,” in CHASE, 2011, pp. 32–39.

Digital Library

[11]

P. Pirolli, Information Foraging Theory: Adaptive Interaction with Information. Oxford University Press, 2007.

Digital Library

[12]

D. W. Stephens and J. R. Krebs, Foraging Theory. Princeton University Press, 1986.

[13]

J. Lawrance, R. Bellamy, and M. Burnett, “Scents in programs: does information foraging theory apply to program maintenance?” in VL/HCC, 2007, pp. 15–22.

Digital Library

[14]

J. Lawrance, R. Bellamy, M. Burnett, and K. Rector, “Using information scent to model the dynamic foraging behavior of programmers in maintenance tasks,” in CHI, 2008, pp. 1323–1332.

Digital Library

[15]

J. Lawrance, M. Burnett, R. Bellamy, C. Bogart, and C. Swart, “Reactive information foraging for evolving goals,” in CHI, 2010, pp. 25–34.

Digital Library

[16]

J. Lawrance, C. Bogart, M. Burnett, R. Bellamy, K. Rector, and S. D. Fleming, “How programmers debug, revisited: an information foraging theory perspective,” IEEE TSE, (accepted).

Digital Library

[17]

C. D. Manning, P. Raghavan, and H. Schütze, Introduction to Information Retrieval. Cambridge University Press, 2008.

Digital Library

[18]

U. Shardanand and P. Maes, “Social information filtering: algorithms for automating “word of mouth”,” in CHI, 1995, pp. 210–217.

Digital Library

[19]

P. Pirolli, “Computational models of information scent-following in a very large browsable text collection,” in CHI, 1997, pp. 3–10.

Digital Library

[20]

E. H. Chi, P. Pirolli, K. Chen, and J. E. Pitkow, “Using information scent to model user information needs and actions on the Web,” in CHI, 2001, pp. 490–497.

Digital Library

[21]

J. R. Anderson and P. Pirolli, “Spread of activation,” Journal of Experimental Psychology, vol. 10, pp. 791–798, 1984.

[22]

H. Schütze, “Dimensions of meaning,” in SC, 1992, pp. 787–796.

Digital Library

[23]

J. M. Spool, C. Perfetti, and D. Brittan, Designing for the Scent of Information. User Interface Engineering, 2004.

[24]

A. J. Ko, B. A. Myers, M. J. Coblenz, and H. H. Aung, “An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks,” IEEE TSE, vol. 32(12), pp. 971– 987, 2006.

Digital Library

[25]

D. Piorkowski, S. D. Fleming, C. Scaffidi, L. John, C. Bogart, B. E. John, M. M. Burnett, and R. K. E. Bellamy, “Modeling programmer navigation: a head-to-head empirical evaluation of predictive models,” in VL/HCC, 2011, pp. 109–116.

[26]

N. Niu, A. Mahmoud, and G. Bradshaw, “Information foraging as a foundation for code navigation,” in ICSE, 2011, pp. 816–819.

Digital Library

[27]

R. Oliveto, M. Gethers, D. Poshyvanyk, and A. De Lucia, “On the equivalence of information retrieval methods for automated traceability link recovery,” in ICPC, 2010, pp. 68–71.

Digital Library

[28]

J. H. Hayes and A. Dekhtyar, “A framework for comparing requirements tracing experiments,” IJSEKE, vol. 15(5), pp. 751–782, 2005.

[29]

D. Cuddeback, A. Dekhtyar, J. H. Hayes, J. Holden, and W.-K. Kong, “Towards overcoming human analyst fallibility in the requirements tracing process (NIER Track),” in ICSE, 2011, pp. 860–863.

Digital Library

[30]

J. R. Anderson, The Adaptive Character of Thought. Lawrence Erlbaum Associates, 1990.

[31]

J. Lin, C. C. Lin, J. Cleland-Huang, R. Settimi, J. Amaya, G. Bedford, B. Berenbach, O. B. Khadra, C. Duan, and X. Zou, “Poirot: a distributed tool supporting enterprise-wide automated traceability,” in RE, 2006, pp. 356–357.

Digital Library

[32]

A. Mahmoud and N. Niu, “TraCter: a tool for candidate traceability link clustering,” in RE, 2011, pp. 335–336.

Digital Library

[33]

J. Cleland-Huang, B. Berenbach, S. Clark, R. Settimi, and E. Romanova, “Best practices for automated traceability,” IEEE Computer, vol. 40(6), pp. 27–35, 2007.

Digital Library

[34]

A. Mahmoud and N. Niu, “Source code indexing for automated tracing,” in TEFSE, 2011, pp. 3–9.

Digital Library

[35]

A. De Lucia, R. Oliveto, and G. Tortora, “IR-based traceability recovery processes: an empirical comparison of “one-shot” and incremental processes,” in ASE, 2008, pp. 39–48.

Digital Library

[36]

N. Niu, J. Savolainen, T. Bhowmik, A. Mahmoud, and S. Reddivari, “A framework for examining topical locality in object-oriented software,” in COMPSAC, 2012, pp. 219–224.

Digital Library

[37]

H. Sultanov, J. H. Hayes, and W.-K. Kong, “Application of swarm techniques to requirements tracing,” REJ, vol. 16, pp. 209–226, 2011.

Digital Library

[38]

C. Duan and J. Cleland-Huang, “Clustering support for automated tracing,” in ASE, 2007, pp. 244–253.

Digital Library

[39]

A. Meneely, B. Smith, and L. Williams, “iTrust electronic health care system: a case study,” in Software and Systems Traceability, J. Cleland-Huang, O. Gotel, and A. Zisman, Eds. Springer, 2012.

[40]

W. J. Conover, Practical Nonparametric Statistics. Wiley, 1999.

[41]

N. Niu and A. Mahmoud, “Enhancing candidate link generation for requirements tracing: the cluster hypothesis revisited,” in RE, 2012, pp. 81–90.

Digital Library

[42]

M. A. Hearst, Search User Interfaces. Cambridge University Press, 2009.

Digital Library

[43]

B. J. Jansen, A. Spink, J. Bateman, and T. Saracevic, “Real life information retrieval: a study of user queries on the Web,” ACM SIGIR Forum, vol. 32(1), pp. 5–17, 1998.

Digital Library

[44]

A. Egyed, P. Grünbacher, M. Heindl, and S. Biffl, “Value-based requirements traceability: lessons learned,” in RE, 2007, pp. 115–118.

[45]

A. Egyed, F. Graf, and P. Grünbacher, “Effort and quality of recovering requirements-to-code traces: two exploratory experiments,” in RE, 2010, pp. 221–230.

Digital Library

Cited By

Penney SDodge JAnderson AHilderbrand CSimpson LBurnett M(2021)The Shoutcasters, the Game Enthusiasts, and the AI: Foraging for Explanations of Real-time Strategy PlayersACM Transactions on Interactive Intelligent Systems10.1145/339604711:1(1-46)Online publication date: 15-Mar-2021
https://dl.acm.org/doi/10.1145/3396047
Dodge JPenney SHilderbrand CAnderson ABurnett MMandryk RHancock MPerry MCox A(2018)How the Experts Do ItProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3174136(1-12)Online publication date: 21-Apr-2018
https://dl.acm.org/doi/10.1145/3173574.3174136
Borg MAlégroth ERuneson PScanniello GLo DSerebrenik A(2017)Software engineers' information seeking behavior in change impact analysisProceedings of the 25th International Conference on Program Comprehension10.1109/ICPC.2017.20(12-22)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.1109/ICPC.2017.20
Show More Cited By

Index Terms

Departures from optimality: understanding human analyst's information foraging in assisted requirements tracing
1. Software and its engineering
  1. Software creation and management
    1. Designing software
      1. Requirements analysis
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools

Recommendations

On human analyst performance in assisted requirements tracing: Statistical analysis
RE '11: Proceedings of the 2011 IEEE 19th International Requirements Engineering Conference

Assisted requirements tracing is a process in which a human analyst validates candidate traces produced by an automated requirements tracing method or tool. The assisted requirements tracing process splits the difference between the commonly applied ...
Berrypicking and information foraging

Exploratory search is a specialised form of information searching behaviour that is open-ended, dynamic and multi-faceted in nature. The study elaborates the picture of exploratory search by concentrating on two theoretical frameworks for studying ...
Towards overcoming human analyst fallibility in the requirements tracing process (NIER track)
ICSE '11: Proceedings of the 33rd International Conference on Software Engineering

Our research group recently discovered that human analysts, when asked to validate candidate traceability matrices, produce predictably imperfect results, in some cases less accurate than the starting candidate matrices. This discovery radically changes ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '13: Proceedings of the 2013 International Conference on Software Engineering

May 2013

1561 pages

ISBN:9781467330763

General Chair:
David Notkin,
Program Chairs:
Betty H. C. Cheng,
Klaus Pohl

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

IEEE Press

Publication History

Published: 18 May 2013

Check for updates

Qualifiers

Research-article

Conference

ICSE '13

Sponsor:

SIGSOFT

ICSE '13: 35th International Conference on Software Engineering

May 18 - 26, 2013

CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
293
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Penney SDodge JAnderson AHilderbrand CSimpson LBurnett M(2021)The Shoutcasters, the Game Enthusiasts, and the AI: Foraging for Explanations of Real-time Strategy PlayersACM Transactions on Interactive Intelligent Systems10.1145/339604711:1(1-46)Online publication date: 15-Mar-2021
https://dl.acm.org/doi/10.1145/3396047
Dodge JPenney SHilderbrand CAnderson ABurnett MMandryk RHancock MPerry MCox A(2018)How the Experts Do ItProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3174136(1-12)Online publication date: 21-Apr-2018
https://dl.acm.org/doi/10.1145/3173574.3174136
Borg MAlégroth ERuneson PScanniello GLo DSerebrenik A(2017)Software engineers' information seeking behavior in change impact analysisProceedings of the 25th International Conference on Program Comprehension10.1109/ICPC.2017.20(12-22)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.1109/ICPC.2017.20
Khatwani CJin XNiu NKoshoffer ANewman LSavolainen J(2017)Advancing viewpoint merging in requirements engineeringRequirements Engineering10.1007/s00766-017-0271-022:3(317-338)Online publication date: 1-Sep-2017
https://dl.acm.org/doi/10.1007/s00766-017-0271-0
Javed MStevanetic SZdun UPreschern CEloranta V(2016)Towards a pattern language for construction and maintenance of software architecture traceability linksProceedings of the 21st European Conference on Pattern Languages of Programs10.1145/3011784.3011810(1-20)Online publication date: 6-Jul-2016
https://dl.acm.org/doi/10.1145/3011784.3011810
Jin XNiu NWagner MBegel ACalefato FTreude C(2016)On the impact of social network information diversity on end-user programming productivity: a foraging-theoretic studyProceedings of the 8th International Workshop on Social Software Engineering10.1145/2993283.2993284(15-21)Online publication date: 14-Nov-2016
https://dl.acm.org/doi/10.1145/2993283.2993284
Niu NWang WGupta AZimmermann TCleland-Huang JSu Z(2016)Gray links in the use of requirements traceabilityProceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering10.1145/2950290.2950354(384-395)Online publication date: 1-Nov-2016
https://dl.acm.org/doi/10.1145/2950290.2950354
Piorkowski DHenley ANabi TFleming SScaffidi CBurnett MZimmermann TCleland-Huang JSu Z(2016)Foraging and navigations, fundamentally: developers' predictions of value and costProceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering10.1145/2950290.2950302(97-108)Online publication date: 1-Nov-2016
https://dl.acm.org/doi/10.1145/2950290.2950302
Wang WNiu NLiu HWu YMäder POliveto R(2015)Tagging in assisted tracingProceedings of the 8th International Symposium on Software and Systems Traceability10.5555/2821445.2821448(8-14)Online publication date: 16-May-2015
https://dl.acm.org/doi/10.5555/2821445.2821448
Bhowmik TNiu NSinghania PWang W(2015)On the Role of Structural Holes in Requirements IdentificationACM Transactions on Management Information Systems10.1145/27952356:3(1-30)Online publication date: 13-Sep-2015
https://dl.acm.org/doi/10.1145/2795235

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents