Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3468264.3468582acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

Characterizing search activities on stack overflow

Published: 18 August 2021 Publication History

Abstract

To solve programming issues, developers commonly search on Stack Overflow to seek potential solutions. However, there is a gap between the knowledge developers are interested in and the knowledge they are able to retrieve using search engines. To help developers efficiently retrieve relevant knowledge on Stack Overflow, prior studies proposed several techniques to reformulate queries and generate summarized answers. However, few studies performed a large-scale analysis using real-world search logs. In this paper, we characterize how developers search on Stack Overflow using such logs. By doing so, we identify the challenges developers face when searching on Stack Overflow and seek opportunities for the platform and researchers to help developers efficiently retrieve knowledge. To characterize search activities on Stack Overflow, we use search log data based on requests to Stack Overflow's web servers. We find that the most common search activity is reformulating the immediately preceding queries. Related work looked into query reformulations when using generic search engines and found 13 types of query reformulation strategies. Compared to their results, we observe that 71.78% of the reformulations can be fitted into those reformulation strategies. In terms of how queries are structured, 17.41% of the search sessions only search for fragments of source code artifacts (e.g., class and method names) without specifying the names of programming languages, libraries, or frameworks. Based on our findings, we provide actionable suggestions for Stack Overflow moderators and outline directions for future research. For example, we encourage Stack Overflow to set up a database that includes the relations between all computer programming terminologies shared on Stack Overflow, e.g., method name, data structure name, design pattern, and IDE name. By doing so, Stack Overflow could improve the performance of search engines by considering related programming terminologies at different levels of granularity.

References

[1]
[n.d.]. Catalog Services for the Web (CSW). https://docs.geoserver.org/stable/en/user/services/csw/index.html
[2]
[n.d.]. CREATE VIEW (Transact-SQL). https://docs.microsoft.com/en-us/sql/t-sql/statements/create-view-transact-sql?view=sql-server-ver15
[3]
[n.d.]. CSW. https://en.wikipedia.org/wiki/CSW
[4]
[n.d.]. How do I ask a good question? https://stackoverflow.com/help/how-to-ask
[5]
[n.d.]. How do I search? https://stackoverflow.com/help/searching
[6]
[n.d.]. How do I write a good answer? https://stackoverflow.com/help/how-to-answer
[7]
[n.d.]. PHP event on session end? https://stackoverflow.com/questions/7226145/php-event-on-session-end
[8]
[n.d.]. reformulationClassifier. http://jeffhuang.com/reformulationClassifier.py
[9]
[n.d.]. Search. https://stackoverflow.com/search
[10]
[n.d.]. SESSIONTIMEZONE. https://docs.oracle.com/database/121/SQLRF/functions176.htm##SQLRF06105
[11]
[n.d.]. Stack Exchange Data Explorer. https://data.stackexchange.com/
[12]
[n.d.]. Total number of questions and answers for the last 12 months (in 30 day chunks). https://data.stackexchange.com/stackoverflow/query/6134/total-questions-and-answers-per-month-for-the-last-12
[13]
[n.d.]. What are tags, and how should I use them? https://stackoverflow.com/help/tagging
[14]
[n.d.]. Why are some questions marked as duplicate? https://stackoverflow.com/helpcenter/duplicates
[15]
Hervé Abdi. 2007. Bonferroni and Šidák corrections for multiple comparisons. Encyclopedia of measurement and statistics, 3 (2007), 103–107.
[16]
Rakesh Agarwal and Ramakrishnan Srikant. 1994. Fast algorithms for mining association rules. In Proc. of the 20th VLDB Conference. 487, 499.
[17]
Rakesh Agrawal, Tomasz Imieliński, and Arun Swami. 1993. Mining association rules between sets of items in large databases. In Proceedings of the 1993 ACM SIGMOD international conference on Management of data. 207–216. https://doi.org/10.1145/170036.170072
[18]
Martin Arlitt. 2000. Characterizing web user sessions. ACM SIGMETRICS Performance Evaluation Review, 28, 2 (2000), 50–63. https://doi.org/10.1145/362883.362920
[19]
Sushil Krishna Bajracharya and Cristina Videira Lopes. 2012. Analyzing and mining a code search engine usage log. Empirical Software Engineering, 17, 4 (2012), 424–466. https://doi.org/10.1007/s10664-010-9144-6
[20]
Alexey Borisov, Ilya Markov, Maarten de Rijke, and Pavel Serdyukov. 2016. A Context-aware Time Model for Web Search. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval - SIGIR ’16. ACM Press, Pisa, Italy. 205–214. isbn:978-1-4503-4069-4 https://doi.org/10.1145/2911451.2911504
[21]
Andrei Z. Broder, Marcus Fontoura, Evgeniy Gabrilovich, Amruta Joshi, Vanja Josifovski, and Tong Zhang. 2007. Robust classification of rare queries using web knowledge. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR ’07. ACM Press, Amsterdam, The Netherlands. 231. isbn:978-1-59593-597-7 https://doi.org/10.1145/1277741.1277783
[22]
Peter Bruza and Simon Dennis. 1997. Query Reformulation on the Internet: Empirical Data and the Hyperindex Search Engine. In RIAO. 97, 488–499.
[23]
Kaibo Cao, Chunyang Chen, Sebastian Baltes, Christoph Treude, and Xiang Chen. 2021. Automated Query Reformulation for Efficient Search based on Query Logs From Stack Overflow. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 1273–1285. https://doi.org/10.1109/ICSE43902.2021.00116
[24]
2020. Query Understanding for Search Engines, Yi Chang and Hongbo Deng (Eds.) (The Information Retrieval Series, Vol. 46). Springer International Publishing, Cham. isbn:978-3-030-58333-0 978-3-030-58334-7 https://doi.org/10.1007/978-3-030-58334-7
[25]
Olivier Chapelle and Ya Zhang. 2009. A dynamic bayesian network click model for web search ranking. In Proceedings of the 18th international conference on World wide web. 1–10. https://doi.org/10.1145/1526709.1526711
[26]
Norman Cliff. 2014. Ordinal methods for behavioral data analysis. Psychology Press. https://doi.org/10.4324/9781315806730
[27]
Olive Jean Dunn. 1961. Multiple comparisons among means. Journal of the American statistical association, 56, 293 (1961), 52–64. https://doi.org/10.1080/01621459.1961.10482090
[28]
Sonal Gupta, Diana L MacLean, Jeffrey Heer, and Christopher D Manning. 2014. Induced lexico-syntactic patterns improve information extraction from online medical forums. Journal of the American Medical Informatics Association, 21, 5 (2014), 902–909. https://doi.org/10.1136/amiajnl-2014-002669
[29]
Matthias Hagen, Martin Potthast, Anna Beyer, and Benno Stein. 2012. Towards optimum query segmentation: in doubt without. In Proceedings of the 21st ACM international conference on Information and knowledge management - CIKM ’12. ACM Press, Maui, Hawaii, USA. 1015. isbn:978-1-4503-1156-4 https://doi.org/10.1145/2396761.2398398
[30]
Daqing He, Ayşe Göker, and David J Harper. 2002. Combining evidence for automatic web session identification. Information Processing & Management, 38, 5 (2002), 727–742. https://doi.org/10.1016/S0306-4573(01)00060-7
[31]
Marti A Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Coling 1992 volume 2: The 15th international conference on computational linguistics. https://doi.org/10.3115/992133.992154
[32]
Sharon Hirsch, Ido Guy, Alexander Nus, Arnon Dagan, and Oren Kurland. 2020. Query reformulation in E-commerce search. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1319–1328. https://doi.org/10.1145/3397271.3401065
[33]
Jeff Huang and Efthimis N. Efthimiadis. 2009. Analyzing and evaluating query reformulation strategies in web search logs. In Proceeding of the 18th ACM conference on Information and knowledge management - CIKM ’09. ACM Press, Hong Kong, China. 77. isbn:978-1-60558-512-3 https://doi.org/10.1145/1645953.1645966
[34]
Bernard J. Jansen, Danielle L. Booth, and Amanda Spink. 2009. Patterns of query reformulation during Web searching. Journal of the American Society for Information Science and Technology, 60, 7 (2009), 1358–1371. issn:1532-2890 https://doi.org/10.1002/asi.21071 _eprint: https://asistdl.onlinelibrary.wiley.com/doi/pdf/10.1002/asi.21071.
[35]
Bernard J Jansen, Amanda Spink, Chris Blakely, and Sherry Koshman. 2007. Defining a session on Web search engines. Journal of the American Society for Information Science and Technology, 58, 6 (2007), 862–871. https://doi.org/10.1002/asi.20564
[36]
Youngho Kim, Ahmed Hassan, Ryen W White, and Imed Zitouni. 2014. Modeling dwell time to predict click-level satisfaction. In Proceedings of the 7th ACM international conference on Web search and data mining. 193–202. https://doi.org/10.1145/2556195.2556220
[37]
William H Kruskal and W Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American statistical Association, 47, 260 (1952), 583–621. https://doi.org/10.1080/01621459.1952.10483441
[38]
Tessa Lau and Eric Horvitz. 1999. Patterns of search: analyzing and modeling web query refinement. In UM99 user modeling. Springer, 119–128. https://doi.org/10.1007/978-3-7091-2490-1_12
[39]
Ying Li, Zijian Zheng, and Honghua (Kathy) Dai. 2005. KDD CUP-2005 report: facing a great challenge. SIGKDD Explor. Newsl., 7, 2 (2005), Dec., 91–99. issn:1931-0145, 1931-0153 https://doi.org/10.1145/1117454.1117466
[40]
Chao Liu, Ryen W White, and Susan Dumais. 2010. Understanding web browsing behaviors through Weibull analysis of dwell time. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. 379–386. https://doi.org/10.1145/1835449.1835513
[41]
Mehdi Manshadi and Xiao Li. 2009. Semantic Tagging of Web Search Queries. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. Association for Computational Linguistics, Suntec, Singapore. 861–869. https://www.aclweb.org/anthology/P09-1097
[42]
M. Nassif, C. Treude, and M. P. Robillard. 2020. Automatically Categorizing Software Technologies. IEEE Transactions on Software Engineering, 46, 1 (2020), 20–32. https://doi.org/10.1109/TSE.2018.2836450
[43]
Liming Nie, He Jiang, Zhilei Ren, Zeyi Sun, and Xiaochen Li. 2016. Query expansion based on crowd knowledge for code search. IEEE Transactions on Services Computing, 9, 5 (2016), 771–783. https://doi.org/10.1109/TSC.2016.2560165
[44]
Cole S. Peterson, Jonathan A. Saddler, Natalie M. Halavick, and Bonita Sharif. 2019. A Gaze-Based Exploratory Study on the Information Seeking Behavior of Developers on Stack Overflow. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland Uk. 1–6. isbn:978-1-4503-5971-9 https://doi.org/10.1145/3290607.3312801
[45]
Ana-Maria Popescu and Orena Etzioni. 2007. Extracting product features and opinions from reviews. In Natural language processing and text mining. Springer, 9–28. https://doi.org/10.1007/978-1-84628-754-1_2
[46]
Filip Radlinski, Madhu Kurup, and Thorsten Joachims. 2008. How does clickthrough data reflect retrieval quality? In Proceedings of the 17th ACM conference on Information and knowledge management. 43–52. https://doi.org/10.1145/1458082.1458092
[47]
Jeanine Romano, Jeffrey D Kromrey, Jesse Coraggio, Jeff Skowronek, and Linda Devine. 2006. Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen’sd indices the most appropriate choices. In annual meeting of the Southern Association for Institutional Research. 1–51.
[48]
Caitlin Sadowski, Kathryn T Stolee, and Sebastian Elbaum. 2015. How developers search for code: a case study. In Proceedings of the 2015 10th joint meeting on foundations of software engineering. 191–201. https://doi.org/10.1145/2786805.2786855
[49]
Dou Shen, Jian-Tao Sun, Qiang Yang, and Zheng Chen. 2006. Building bridges for web query classification. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR ’06. ACM Press, Seattle, Washington, USA. 131. isbn:978-1-59593-369-0 https://doi.org/10.1145/1148170.1148196
[50]
Janice Singer, Timothy Lethbridge, Norman Vinson, and Nicolas Anquetil. 2010. An examination of software engineering work practices. In CASCON First Decade High Impact Papers on - CASCON ’10. ACM Press, Toronto, Ontario, Canada. 174–188. https://doi.org/10.1145/1925805.1925815
[51]
Jun Song, Jun Xiao, Fei Wu, Haishan Wu, Tong Zhang, Zhongfei Mark Zhang, and Wenwu Zhu. 2017. Hierarchical contextual attention recurrent neural network for map query suggestion. IEEE Transactions on Knowledge and Data Engineering, 29, 9 (2017), 1888–1901. https://doi.org/10.1109/TKDE.2017.2700392
[52]
Lynda Tamine, Jesús Lovón Melgarejo, and Karen Pinel-Sauvagnat. 2020. What Can Task Teach Us About Query Reformulations? In European Conference on Information Retrieval. 636–650. https://doi.org/10.1007/978-3-030-45439-5_42
[53]
Raphael Tang, Ferhan Ture, and Jimmy Lin. 2019. Yelling at Your TV: An Analysis of Speech Recognition Errors and Subsequent User Behavior on Entertainment Systems. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 853–856. https://doi.org/10.1145/3331184.3331271
[54]
Jaime Teevan, Eytan Adar, Rosie Jones, and Michael AS Potts. 2007. Information re-retrieval: repeat queries in Yahoo’s logs. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 151–158. https://doi.org/10.1145/1277741.1277770
[55]
Anthony J Viera and Joanne M Garrett. 2005. Understanding Interobserver Agreement: The Kappa Statistic. Family Medicine, 37, 5 (2005), 360–363.
[56]
Martin Whittle, Barry Eaglestone, Nigel Ford, Valerie J Gillet, and Andrew Madden. 2007. Data mining of search engine logs. Journal of the American Society for Information Science and Technology, 58, 14 (2007), 2382–2400. https://doi.org/10.1002/asi.20733
[57]
Xin Xia, Lingfeng Bao, David Lo, Pavneet Singh Kochhar, Ahmed E Hassan, and Zhenchang Xing. 2017. What do developers search for on the web? Empirical Software Engineering, 22, 6 (2017), 3149–3185. https://doi.org/10.1007/s10664-017-9514-4
[58]
Bowen Xu, Zhenchang Xing, Xin Xia, and David Lo. 2017. AnswerBot: Automated generation of answer summary to developers’ technical questions. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 706–716. https://doi.org/doi/10.5555/3155562.3155650
[59]
Neng Zhang, Qiao Huang, Xin Xia, Ying Zou, David Lo, and Zhenchang Xing. 2020. Chatbot4QR: Interactive Query Refinement for Technical Question Retrieval. IEEE Transactions on Software Engineering, https://doi.org/10.1109/TSE.2020.3016006
[60]
Heinz Züllighoven. 2004. Object-oriented construction handbook: Developing application-oriented software with the tools & materials approach. Elsevier. https://doi.org/10.1016/B978-155860687-6/50005-0

Cited By

View all
  • (2024)What causes exceptions in machine learning applications? Mining machine learning-related stack traces on Stack OverflowEmpirical Software Engineering10.1007/s10664-024-10499-929:5Online publication date: 3-Jul-2024
  • (2023)A Field Study of Developer Documentation FormatExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585767(1-7)Online publication date: 19-Apr-2023
  • (2023)CodeGen4Libs: A Two-Stage Approach for Library-Oriented Code Generation2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00159(434-445)Online publication date: 11-Sep-2023
  • Show More Cited By

Index Terms

  1. Characterizing search activities on stack overflow

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
    August 2021
    1690 pages
    ISBN:9781450385626
    DOI:10.1145/3468264
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 18 August 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Data Mining
    2. Query Logs
    3. Query Reformulation
    4. Stack Overflow

    Qualifiers

    • Research-article

    Funding Sources

    • Key Research and Development Program of Zhejiang Province
    • National Science Foundation of China
    • the National Research Foundation, Singapore under its Industry Alignment Fund ? Prepositioning (IAF-PP) Funding Initiative.

    Conference

    ESEC/FSE '21
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 112 of 543 submissions, 21%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)60
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 23 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)What causes exceptions in machine learning applications? Mining machine learning-related stack traces on Stack OverflowEmpirical Software Engineering10.1007/s10664-024-10499-929:5Online publication date: 3-Jul-2024
    • (2023)A Field Study of Developer Documentation FormatExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585767(1-7)Online publication date: 19-Apr-2023
    • (2023)CodeGen4Libs: A Two-Stage Approach for Library-Oriented Code Generation2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00159(434-445)Online publication date: 11-Sep-2023
    • (2022)SOTitle: A Transformer-based Post Title Generation Approach for Stack Overflow2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00075(577-588)Online publication date: Mar-2022
    • (2022)Time to vote: Temporal clustering of user activity on Stack OverflowJournal of the Association for Information Science and Technology10.1002/asi.2465873:12(1681-1691)Online publication date: 21-May-2022

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media